[Python-Dev] writelines() not thread-safe

[Guido van Rossum]

Christian Tismer just did an exhaustive search for thread unsafe use
of Python operations, and found two weaknesses.  One is
posix.listdir(), which I had already found; the other is
file.writelines().  Here's a program that demonstrates the bug;
basically, while writelines is walking down the list, another thread
could truncate the list, causing PyList_GetItem() to fail or a string
object to be deallocated while writelines is using it.  On my SOlaris
7 system it typically crashes in the first or second iteration.

It's easy to fix: just don't use release the interpreter lock (get rid
of Py_BEGIN_ALLOW_THREADS c.s.).  This would however prevent other
threads from doing any work while this thread may be blocked for I/O.

An alternative solution is to put Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS just around the fwrite() call.  This is safe, but
would require a lot of lock operations and would probably slow things
down too much.

Ideas?

--Guido van Rossum (home page: http://www.python.org/~guido/)
import os
import sys
import thread
import random
import time
import tempfile

def good_guy(fp, list):
    t0 = time.time()
    fp.seek(0)
    fp.writelines(list)
    t1 = time.time()
    print fp.tell(), "bytes written"
    return t1-t0

def bad_guy(dt, list):
    time.sleep(random.random() * dt)
    del list[:]

def main():
    infn = "/usr/dict/words"
    if sys.argv[1:]:
        infn = sys.argv[1]
    print "reading %s..." % infn
    fp = open(infn)
    list = fp.readlines()
    fp.close()
    print "read %d lines" % len(list)
    tfn = tempfile.mktemp()
    fp = None
    try:
        fp = open(tfn, "w")
        print "calibrating..."
        dt = 0.0
        n = 3
        for i in range(n):
            dt = dt + good_guy(fp, list)
        dt = dt / n # average time it took to write the list to disk
        print "dt =", round(dt, 3)
        i = 0
        while 1:
            i = i+1
            print "test", i
            copy = map(lambda x: x[1:], list)
            thread.start_new_thread(bad_guy, (dt, copy))
            good_guy(fp, copy)
    finally:
        if fp:
            fp.close()
        try:
            os.unlink(tfn)
        except os.error:
            pass

main()