Serializing PyThreads without holding interpreter lock?

[Albert Hofkamp]

Hello all,

A colleague of mine has written a Python program that uses threads, one
thread is used for the GUI, the other for performing computations.
The latter uses an external solver, written in C.

The interface to the solver has been written in C to connect it to Python.
The interface also allows call-back functions written in Python to be
called from the solver.

The program runs fine, except when the computation thread calls the solver.
At that moment, Python calls the C function which performs it computations
for a minute or so. During this time, the computation thread holds the
global interpreter lock, and the GUI thread cannot update the user
interface.


At first sight, a solution to this problem seems to be releasing the
global interpreter lock, by
1) adding Py_BEGIN_UNBLOCK_THREADS and Py_BEGIN_BLOCK_THREADS around the
   call to the solver, and
2) adding Py_BEGIN_BLOCK_THREADS and Py_BEGIN_UNBLOCK_THREADS around the
   call-back interface code just before and after the Py_call_func() call.
(and pass the PyThreadState around through the solver, from the solver call
to the call-back code.)

However, releasing the global interpreter lock makes the solver re-entrant,
i.e. a second thread could call the solver while the first thread is still
computing. The solver library however cannot handle this situation, and
will produce senseless results (so I assume).

The question is now how to prevent this from happening, i.e. how do I
serialize calls to the solver without holding the global interpreter lock?
A nice solution would be to block the second thread until the first thread
is finished, e.g. by putting a P() and V() operation around the call to the
solver.
The Python C programming manuals do give a number of primitives for
handling thread, but blocking threads is not part of the provided
functionality.


Albert
-- 
Unlike popular belief, the .doc format is not an open publically available format.