flock seems very unsafe, python fcntl bug?

[Nick Craig-Wood]

Jens Henrik Leonhard Jensen <jhlj at statsbiblioteket.dk> wrote:
>  Your problem is that open(...,'w') is not locked.
> 
>  Use something like:
>  lockf = open('aaa', 'a')
>  fnctl.flock(lockf,fnctl.LOCK_EX)
>  file = open('aaa', 'w')
>  file.write('asdf')
>  file.close()
>  lockf.close()

I've not seen that trick before - it is a good one.  It wouldn't work
for mandatory locking though...

The basic problem is getting the file open for read/write without
destroying the contents before you have the lock.  None of the
arguments to open() do the right thing....

Using an append open is a nice solution to that, apart from the fact
that you can only append data onto the file hence the other open.

Here are some other possible solutions and a test harness!  The low
level open is probably the most unixy solution.  If you could specify
os.O_RDWR|os.O_CREAT to the python open() function then it would be
the best, but none of the documented open strings do that

$ strace python -c 'open("aaa", "r+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_LARGEFILE)         = 3
$ strace python -c 'open("aaa", "w+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
$ strace python -c 'open("aaa", "wr+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
$ strace python -c 'open("aaa", "a+")' 2>&1 | grep 'open("aaa"'
open("aaa", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 3


------------------------------------------------------------
"""
Testing writing stuff to a file with locking
"""

import os
import sys
import fcntl

def method_0(data):
    """
    Use an normal open

    This doesn't work because it truncates the data before it has the
    lock
    """
    fd = open('aaa', 'w')
    fcntl.flock(fd,fcntl.LOCK_EX)
    fd.write(data)
    fd.close()

def method_1(data):
    """
    Use an additional append open to lock
    """
    lock_fd = open('aaa', 'a')
    fcntl.flock(lock_fd,fcntl.LOCK_EX)
    fd = open('aaa', 'w')
    fd.write(data)
    fd.close()
    lock_fd.close()

def method_2(data):
    """
    Use a low level open
    """
    fd = os.fdopen(os.open('aaa', os.O_CREAT|os.O_RDWR), "r+")
    fcntl.flock(fd, fcntl.LOCK_EX)
    fd.truncate()
    fd.write(data)
    fd.close()

def method_3(data):
    """
    Using high level functions only

    Possibly racy on open when file doesn't exist
    """
    if not os.path.exists('aaa'):
        fd = open('aaa', 'w+')
    else:
        fd = open('aaa', 'r+')
    fcntl.flock(fd,fcntl.LOCK_EX)
    fd.truncate()
    fd.write(data)
    fd.close()

def check():
    fd = open('aaa', 'r')
    fcntl.flock(fd,fcntl.LOCK_EX)
    data = fd.read()
    fd.close()
    if data not in ("sausage", "potato"):
        raise AssertionError("Wrong data %r" % data)

if os.path.exists("aaa"):
    os.unlink("aaa")

if len(sys.argv) < 2:
    print "Syntax: %s <method number 0..3>" % sys.argv[0]
    raise SystemExit(1)
method = globals()["method_"+sys.argv[1]]
if os.fork():
    while 1:
        method("sausage")
        check()
else:
    while 1:
        method("potato")
        check()
------------------------------------------------------------


-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick