How to safely maintain a status file

[Steven D'Aprano]

On Thu, 12 Jul 2012 21:26:20 -0700, rantingrickjohnson wrote:

> On Thursday, July 12, 2012 10:13:47 PM UTC-5, Steven D'Aprano wrote:
>> Rick has obviously never tried to open a file for reading when somebody
>> else has it opened, also for reading, and discovered that despite
>> Windows being allegedly a multi-user operating system, you can't
>> actually have multiple users read the same files at the same time.
> 
> You misread my response. My comment was direct result of Christian
> stating:
> 
> (paraphrase) "On some systems you are not permitted to delete a file
> whilst the file is open "
> 
> ...which seems to be consistent to me. Why would *anybody* want to
> delete a file whilst the file is open? 

Because it is useful and a sensible thing to do.

Why should one misbehaved application, keeping a file open, be allowed to 
hold every other application, and the file system, hostage?

This is one of the many poor decisions which makes Windows so vulnerable 
to viruses and malware. If malware can arrange to keep itself open, you 
can't delete it. Thanks guys!


> Bringing back the car analogy
> again: Would you consider jumping from a moving vehicle a consistent
> interaction with the interface of a vehicle? Of course not. The
> interface for a vehicle is simple and consistent:
> 
>  1. You enter the vehicle at location A 
>  2. The vehicle transports you to location B 
>  3. You exit the vehicle

Amusingly, you neglected to specify "the vehicle stops" -- and rightly 
so, because of course having to stop the vehicle is not a *necessary* 
condition for exiting it, as tens of thousands of stunt men and women can 
attest.

Not to mention people parachuting out of an airplane, pirates or 
commandos boarding a moving ship, pedestrians transferring from a slow 
moving walkway to a faster moving walkway, farmers jumping off a trailer 
while it is still being towed behind a tractor (and jumping back on 
again), and Bruce Willis in "Red" in very possibly the best slow-motion 
action sequence in the history of Hollywood.

http://www.youtube.com/watch?v=xonMpj2YyDU


> At no time during the trip would anyone expect you to leap from the
> vehicle. 

Expected or not, you can do so.


> But when you delete open files, you are essentially leaping
> from the moving vehicle! This behavior goes against all expectations of
> consistency in an API -- and against all sanity when riding in a
> vehicle!

Fortunately, files on a file system are not cars, and deleting open files 
is a perfectly reasonable thing to do, no more frightening than in Python 
deleting a reference to an object using the del statement. Imagine how 
stupid it would be if this happened:


py> x = 42
py> y = x
py> del y
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
DeleteError: cannot delete reference to object '42' until no other 
references to it exist


Fortunately, Python doesn't do that -- it tracks when the object is no 
longer being accessed, and only then physically reclaims the memory used. 
And so it is on POSIX file systems: the file system keeps track of when 
the file on disk is no longer being accessed, and only then physically 
reclaims the blocks being used. Until then, deleting the file merely 
unlinks the file name from the blocks on disk, in the same way that 
"del y" merely unlinks the name y from the object 42.


>> Opening files for exclusive read *by default* is a pointless and silly
>> limitation. It's also unsafe: if a process opens a file for
>> exclusive read, and then dies, *no other process* can close that file.
> 
> Oh come on. Are you actually going to use "errors" or "unintended
> consequences", or even "Acts of God" to defend your argument? 

Features have to be judged by their actual consequences, not some 
unrealistic sense of theoretical purity. The actual consequences of 
mandatory exclusive file locking is, *it sucks*.

Windows users are used to having to reboot their server every few days 
because something is broken, so they might not mind rebooting it because 
some file is locked in a mandatory open state and not even the operating 
system can unlock it. But for those with proper operating systems who 
expect months of uninterrupted service, mandatory locking is a problem to 
be avoided, not a feature.


> Okay.
> Okay. I suppose "IF" the car spontaneously combusted "THEN" the
> passengers would be wise to jump out, leaving the vehicle to the whims
> of inertia.

In this analogy, is the car the file name, the inode, or the directory? 
Are the passengers the file name(s), or the file contents, or the inode? 
Is the driver meant to be the file system? If I have a hard link to the 
file, does that mean the passengers are in two cars at once, or two lots 
of passengers in the same car?


>> One neat trick is to open a file, then delete it from disk while it is
>> still open. So long as your process is still running, you can write to
>> this ghost file, as normal, but no other process can (easily) see it.
>> And when your process ends, the file contents is automatically deleted.
> 
> Well "neat tricks" aside, I am of the firm belief that deleting files
> should never be possible whilst they are open.

[condescension = ON]

Good for you Rick. Having strongly held opinions on things you have only 
a limited understanding about is your right as an American.

[condescension = OFF]


>  * Opening files requires that data exist on disk 
>  * Reading and writing files requires an open file obj 

You have missed a step in jumping from files on disk to open file objects.

Open file objects do not necessarily correspond to files on disk. For 
example, in standard Pascal, file objects are purely in-memory constructs 
emulating files on a tape drive, with no relationship to on-disk files.

(Any half-decent Pascal compiler or interpreter will *also* give you ways 
to access real files on disk, but that isn't covered by the standard.)

Even when the file object does come from an actual disk file, we can 
conclude that before you can open a file for reading, it must exist; but 
having opened it, there is no *necessary* requirement that it *remains* 
on disk. If you try to read from an open file object whose underlying 
file has been deleted, there are three perfectly reasonable behaviours:

- you get an error;

- it is the same result as if the file has been truncated to zero bytes;

- deleting the file only deletes the *name*, not contents, until the 
  last open file handle is shut, and then the contents are deleted.


>  * Closing files requires an open file object 

Naturally; but open file objects don't require that the on-disk file 
still exists.


>  * And deleting files requires that the file NOT be open

Not at all.


> Would you also entertain the idea of reading or writing files that do
> not exist? (not including pseudo file objs like StringIO of course!).

Define "file" and "exist". Because you are conflating at least three 
different things:

a file name
an inode (blocks on a disk)
a file object

Of course it is useful to break the abstraction that file objects must be 
files on disk. StringIO is one such example. Standard Pascal file objects 
is another. Likewise, it is useful to be able to read from an inode that 
is no longer connected to a file name.

So, absolutely, yes, it is useful to be able to read and write from files 
that don't exist, under some circumstances.


> Summary: Neat tricks and Easter eggs are real hoot, but consistency 
> in APIs is the key.

A foolish consistency is the hobgoblin of little minds.


-- 
Steven