[Chicago] help with file locking

[Dan McGee]

On Wed, Feb 29, 2012 at 11:50 AM, Massimo Di Pierro
<mdipierro at cs.depaul.edu> wrote:
> Hello everybody. I am stuck with a problem:
>
> ===== begin test.py =====
> #!/usr/bin/env python
>
>
> # -*- coding: utf-8 -*-
>
>
> import fcntl
> import multiprocessing
> import unittest
>
> N=10000
>
> def lock(fp, mode):
>     fcntl.flock(fp.fileno(), mode)
>
> def unlock(fp):
>     fcntl.flock(fp.fileno(), fcntl.LOCK_UN)
You can pass fp directly to these, they handle file-like objects.

>
> def read_write(args):
>     (filename, iterations) = args
>     for i in range(0, iterations):
>         fp = open(filename,'r')
>         lock(fp,fcntl.LOCK_SH) # shared lock
>         content = fp.read()
>         unlock(fp)             # unlock
>         fp.close()
>         if len(content)!=N:
>             return False
>         fp = open(filename,'w')
You're violating the rules of opening files. From the docs:
    'w' for writing (truncating the file if it already exists)

Simple as that.

Instead, you need something like this for your write section:

        fp = open(filename, 'a')
        lock(fp, fcntl.LOCK_EX, pn) # exclusive lock
        fp.seek(0)
        fp.truncate()
        fp.write(content)
        fp.flush()
        unlock(fp, pn)             # unlock
        fp.close()

>         lock(fp,fcntl.LOCK_EX) # exclusive lock
>         fp.write(content)
You need a call to `fp.flush()` here for sure- Python is buffering
your writes and isn't guaranteed to do anything until you call
fp.close(), after your unlock.

>         unlock(fp)             # unlock
>         fp.close()
>     return True
>
> class TestParallelLocks(unittest.TestCase):
>
>     def setUp(self):
>         self.filename = 'test.txt'
>         contents = 'x'*N
>         fp = open(self.filename,'w')
>         fp.write(contents)
>         fp.close()
>
>     def tearDown(self):
>         try:
>             os.remove(self.filename)
>         except:
>             pass
>
>     def test_reads_and_writes(self):
>         readwriters = 10
>         pool = multiprocessing.Pool(processes = readwriters)
>         results = pool.map(read_write, [[self.filename, 10]] * readwriters)
>         for result in results:
>             self.assertTrue(result)
>
> if __name__ == '__main__':
>     unittest.main()
> ====== end test.py =====
>
> When I run it, it will often fail the test (on Leopard, Lion, Ubuntu, Python
> 2.5, 2.6, 2.7).
>
> Can you reproduce the problem? Do you see anything wrong with the code?
>
> Massimo
>
>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>