When will os.remove fail?

[Chris Angelico]

On Tue, Mar 14, 2017 at 10:32 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Mon, 13 Mar 2017 08:47 pm, eryk sun wrote:
>> One hurdle to getting delete access is the sharing mode. If there are
>> existing File objects that reference the file, they all have to share
>> delete access. Otherwise the open fails with a sharing violation. This
>> is often the show stopper because the C runtime (and thus CPython,
>> usually) opens files with read and write sharing but not delete
>> sharing.
>
> This is the famous "can't delete a file which is open under Windows"
> problem, am I right?
>
> I take it that you *can* delete open files, but only if the process that
> opens them takes special care to use "delete sharing". Is that correct?

Yes, but you can't always control the process that opens them. For
example, it's annoyingly difficult to update a running executable.

>> In user mode, a kernel object such as a File instance is referenced as
>> a handle.
>
> Out of curiosity, I only know the term "handle" from classic Macintosh
> (pre-OS X) where a handle was a managed pointer to a pointer to a chunk of
> memory. Being managed, the OS could move the memory around without the
> handles ending up pointing to garbage. Is that the same meaning in Windows
> land?

The old Mac meaning was very concrete and specific: it's a pointer to
a pointer. Since all *you* ever see is the first pointer, the OS is
free to (atomically) move the destination.

More generally, a handle is just an opaque cookie that represents a
thing. If you do cross-platform socket programming, you'll find that
on Unix, creating a socket returns a "file descriptor"; but on
Windows, it returns a "socket handle". Aside from the fact that socket
handles can only be used with socket functions (making them harder to
pass to other processes etc), the two concepts work broadly the same
way: you have a small integer value that represents a large resource.

>> When the delete disposition is set, no new File references can be
>> instantiated (i.e. any level of access is denied, even just to read
>> the file attributes), but the file isn't immediately unlinked. It's
>> still listed in the parent directory. Any existing File reference that
>> has delete access can unset the delete disposition.
>>
>> When all handle and pointer references to a File object are closed,
>> the file system decrements the reference count on the FCB. When the
>> FCB reference count drops to 0, if the delete disposition is currently
>> set, then finally the file is unlinked from the parent directory.
>
> So technically file deletions aren't atomic in Windows? In principle, I
> could say:
>
> delete file X
>
> which then returns immediately, and if I try to open(X) it will fail. But I
> can still see it if I do a dir() on the parent directory?
>
> Eventually the last reference to X will go away, and then it is unlinked.
> What happens if I pull the plug in the meantime? Will the file magically
> come back on rebooting?

Without actually testing it, my answers (based on experience of using
Windows) would be that the file doesn't disappear at all until it's
actually deleted. The "no new File refs can be instantiated" would
mean that it kicks back an error, not that it looks like the file
doesn't exist. If you pull the plug, it won't "magically come back" -
it'll have never gone.

> That's intended to emulate the behaviour on Unix where deleting a file that
> you own will succeed even if you don't have write access to the file.
>
> (The bash rm command will ask you before deleting, but Python's os.remove
> just removes it.)

(And the rm command won't ask if you say "-f".)

> Does this seem reasonable? Or overly complicated for not enough benefit?

I'm terrified to think what the interactions would be between this and
the Ubuntu subsystem in Win 10. What're the semantics of unlinkat()?
What about the common practice of creating a file and then unlinking
it so it doesn't show up in the directory? (And is that different on a
file system that supports hard links?) Or are heaps of POSIX features
simply not available?

ChrisA