[Python-Dev] BEGIN_ALLOW_THREADS

[Guido van Rossum]

> One of the functions (say foo()) in my extension module calls another
> function (real_foo()) which might block since it does creepy things like
> I/O.  Being a nice fella I surrounded the call with Py_BEGIN_ALLOW_THREADS
> and Py_END_ALLOW_THREADS so that other threads could jump in and do stuff.
> 
> The problem is that real_foo() might call a previously defined callback
> (written in C) which in turn has to call a Python callback (what good is
> an extension module if you have to write your callbacks in C?), but when I
> try to PyEval_CallObject() it - KABOOM!  Segfault somewhere in the
> interpreter (on PyFrame_New() or something like that).
> 
> After a lot of thinking (and a lot of help from python-list) I figured out
> that the state I push away using Py_BEGIN_ALLOW_THREADS must be retrieved
> again before the callback is called, or PyFrame_New() won't like me.
> Problem is that that state variable is declared in Py_BEGIN_ALLOW_THREADS
> and there's no way to reach it (and it also includes a naughty { with the
> corresponding } in the Py_END_ALLOW_THREADS so it won't work to use them
> anyway).
> 
> My solution: I wrote the two macros above.  This works for me, but
> obviously I have to update my macros when the "real" macros in the Python
> distribution change.

Instead of asking us to add more macros, I recommend that you use the
lower-level C API that is used by the macros.  There's a bunch of ways
to save and restore the state.  The macros are just shortcuts (and a
handy way to make all the code disappear when threads don't exist).

--Guido van Rossum (home page: http://www.python.org/~guido/)