How to safely maintain a status file

[Steven D'Aprano]

On Fri, 13 Jul 2012 12:12:01 +1000, Chris Angelico wrote:

> On Fri, Jul 13, 2012 at 11:20 AM, Rick Johnson
> <rantingrickjohnson at gmail.com> wrote:
>> On Jul 12, 2:39 pm, Christian Heimes <li... at cheimes.de> wrote:
>>> Windows's file system layer is not POSIX compatible. For example you
>>> can't remove or replace a file while it is opened by a process.
>>
>> Sounds like a reasonable fail-safe to me.

Rick has obviously never tried to open a file for reading when somebody 
else has it opened, also for reading, and discovered that despite Windows 
being allegedly a multi-user operating system, you can't actually have 
multiple users read the same files at the same time.

(At least not unless the application takes steps to allow it.)

Or tried to back-up files while some application has got them opened. Or 
open a file while an anti-virus scanner is oh-so-slooooowly scanning it.

Opening files for exclusive read *by default* is a pointless and silly 
limitation. It's also unsafe: if a process opens a file for exclusive 
read, and then dies, *no other process* can close that file.

At least on POSIX systems, not even root can override a mandatory 
exclusive lock (it would be pretty pointless if it could), so a rogue or 
buggy program could wreck havoc with mandatory exclusive file locks. 
That's why Linux, by default, treats exclusive file locks as advisory 
(cooperative), not mandatory.

In general, file locking is harder than it sounds, with many traps for 
the unwary, and of course the semantics are dependent on both the 
operating system and the file system.

https://en.wikipedia.org/wiki/File_locking


> POSIX says that files and file names are independent. I can open a file
> based on its name, delete the file based on its name, and still have the
> open file there. When it's closed, it'll be wiped from the disk.

One neat trick is to open a file, then delete it from disk while it is 
still open. So long as your process is still running, you can write to 
this ghost file, as normal, but no other process can (easily) see it. And 
when your process ends, the file contents is automatically deleted.

This is remarkably similar to what Python does with namespaces and dicts:

# create a fake "file system"
ns = {'a': [], 'b': [], 'c': []}
# open a file
myfile = ns['a']
# write to it
myfile.append('some data')
# delete it from the "file system"
del ns['a']
# but I can still read and write to it
myfile.append('more data')
print(myfile[0])
# but anyone else will get an error if they try
another_file = ns['a']


-- 
Steven