[Python-Dev] writelines() not thread-safe

[Tim Peters]

[Guido van Rossum]
> Christian Tismer just did an exhaustive search for thread unsafe use
> of Python operations, and found two weaknesses.  One is
> posix.listdir(), which I had already found; the other is
> file.writelines().  Here's a program that demonstrates the bug;
> basically, while writelines is walking down the list, another thread
> could truncate the list, causing PyList_GetItem() to fail or a string
> object to be deallocated while writelines is using it.  On my SOlaris
> 7 system it typically crashes in the first or second iteration.
>
> It's easy to fix: just don't use release the interpreter lock (get rid
> of Py_BEGIN_ALLOW_THREADS c.s.).  This would however prevent other
> threads from doing any work while this thread may be blocked for I/O.
>
> An alternative solution is to put Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS just around the fwrite() call.  This is safe, but
> would require a lot of lock operations and would probably slow things
> down too much.
>
> Ideas?

2.5:

1: Before releasing the lock, make a shallow copy of the list.

1.5:  As in #1, but iteratively peeling off "the next N" values, for some N
balancing the number of lock operations against the memory burden (I don't
care about the speed of a shallow copy here ...).

2. Pull the same trick list.sort() uses:  make the list object immutable for
the duration (I know you think that's a hack, and it is <wink>, but it costs
virtually nothing and would raise an approriate error when they attempted
the insane mutation).

I actually like #2 best now, but won't in the future, because
file_writelines() should really accept an argument of any sequence type.
This makes 1.5 a better long-term hack.

although-adding-1.5-to-1.6-is-confusing<wink>-ly y'rs  - tim