When will os.remove fail?

[eryk sun]

On Sun, Mar 12, 2017 at 5:48 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
>
> Does os.remove work like this under Windows too?

os.remove calls DeleteFile on Windows. This in turn calls NtOpenFile
to instantiate a kernel File object that has delete access and return
a handle to it. Next it calls NtSetInformationFile on the handle to
set the file's "DeleteFile" disposition. Either of these operations
can fail.

One hurdle to getting delete access is the sharing mode. If there are
existing File objects that reference the file, they all have to share
delete access. Otherwise the open fails with a sharing violation. This
is often the show stopper because the C runtime (and thus CPython,
usually) opens files with read and write sharing but not delete
sharing.

Another hurdle is paging files, but a program really has no business
trying to delete a critical system file. You'll get a sharing
violation if a file is currently mapped as one of the system's paging
files. Windows supports up to 16 paging files, each on a separate
volume, so it's not like this is a common problem.

Otherwise, getting delete access depends on the file system. If it
doesn't implement security (e.g. FAT32), then delete access is always
granted. If it does implement security (e.g. NTFS), then first I think
we need to discuss the basics of NT security.

----

Secured objects have a security descriptor that contains the security
identifier (SID) of the owner and (optionally) a primary group (not
used by Windows). It also may contain a discretionary access control
list (DACL) and a system access control list (SACL). Each of these
contains a list of access control entries (ACEs) that either allow or
deny the specified access mask for a given user/group SID
(canonically, deny ACEs should be sorted before allow ACEs). For use
with container objects, such as file-system directories, an ACE also
has the following flags to control inheritance: object inherit,
container inherit, no propagate inherit (i.e. clear the
object/container inherit flags on inherited ACEs), and inherent only
(i.e. don't apply this ACE to the object itself).

An access mask is a 32-bit value, with one right mapped to each bit.
The lower half defines up to 16 rights that are specific to an object
type, such as File, Process, or Thread. The next 8 bits are standard
rights (e.g. delete) that are defined the same for all objects. The
upper 4 bits are for generic read (GR), write (GW), execute (GE), and
all (GA) access. In an access check, generic rights should be
translated to standard and specific rights according to a generic
mapping that's defined for each type.

Each process has a primary token, and each thread in a process
optionally can have an impersonation token. An access token (and the
associated logon session) is usually created by the local security
authority (the lsass.exe process), which is coupled with the security
monitor in the kernel. A token contains a list of user/group SIDs,
which are used in an access check if they're flagged as either enabled
or use-for-deny-only (i.e. only for deny ACEs). A token also has a
list of privileges and their enable state. A token is itself a secured
object; it's an instance of the Token type.

A token is assigned an integrity level. The term "elevating" refers in
part to starting a process with a token that has a higher integrity
level, for which the available levels are low, medium (default), high,
and system. A secured object's SACL can contain a mandatory label ACE,
which sets its integrity level. The mask field of this ACE isn't a
32-bit access mask, but rather it consists of up to 3 flag values --
no-write-up, no-read-up, and no-execute-up -- that determine whether a
token at a lower integrity level is allowed write/delete, read, or
execute access. If a secured object doesn't have a mandatory label
ACE, then it implicitly has a medium integrity level with no-write-up.

----

OK, that covers the basics.

If the current token contains the privilege SeRestorePrivilege (e.g. a
token for an elevated administrator), and it's enabled, then delete
access will always be granted regardless of the file's security
descriptor. The restore privilege is quite empowering, so enable it
with caution.

A file's DACL will either allow or explicitly deny delete access. If
access isn't explicitly allowed, then it's implicitly denied. However,
delete access will still be granted if the parent directory grants
delete-child access. The delete-child right is specific to the File
type -- specifically for directories.

If the file's SACL mandatory access includes no-write-up, then a user
at a lower integrity level will not be able to open the file for
delete access, even if the file's DACL (discretionary access)
otherwise allows it. However, delete access is still granted if the
parent directory grants delete-child access to the user. Thus
parent-level discretionary access trumps object-level mandatory
access.

The next step is setting the file's "DeleteFile" disposition. First
let's take a close look at how this works.

----

In user mode, a kernel object such as a File instance is referenced as
a handle. Duplicating the handle creates a new reference to the
object. However, if you open the file/device again, you'll get a
handle for a new object. All File objects that reference the same data
file will share a file or link control block (FCB) structure that,
among other things, is used to track sharing of read, write, and
delete access and the file's delete disposition.

When the delete disposition is set, no new File references can be
instantiated (i.e. any level of access is denied, even just to read
the file attributes), but the file isn't immediately unlinked. It's
still listed in the parent directory. Any existing File reference that
has delete access can unset the delete disposition.

When all handle and pointer references to a File object are closed,
the file system decrements the reference count on the FCB. When the
FCB reference count drops to 0, if the delete disposition is currently
set, then finally the file is unlinked from the parent directory.

----

I can think of a couple of cases in which setting the delete
disposition will fail with access denied.

If the file is currently memory-mapped as code or data (e.g. an EXE or
DLL), but not for system paging, then you can open it for delete
access, but only renaming the file will succeed (it can be renamed
anywhere on the volume, but not to another volume). Setting the delete
disposition will fail with access denied. The system won't allow a
mapped file to be unlinked.

Finally, I'm sure most people are familiar with the read-only file
attribute. If this attribute is set you can still open a file with
delete access to rename it, but setting the delete disposition will
fail with access denied.

You may complain that the system has granted the user delete access,
so getting an access denied error is a breach of contract. You'd be
right, but the kernel actually returns STATUS_CANNOT_DELETE. The
Windows API maps this to ERROR_ACCESS_DENIED.

    STATUS_CANNOT_DELETE = 0xC0000121
    ERROR_ACCESS_DENIED = 0x0005

    >>> ntdll.RtlNtStatusToDosError(STATUS_CANNOT_DELETE)
    5

It's often the case that useful information is lost when mapping a
kernel status code to a Windows error code. This is an inherent
problem with layered APIs. Even more information is lost when the C
runtime maps the Windows error to a POSIX errno value such as EACCES.