[Python-Dev] BEGIN_ALLOW_THREADS

[Martin Sjögren]

On Mon, Jul 23, 2001 at 02:21:57PM +0200, Martin Sj=F6gren wrote:
> Is there a reason the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS
> don't allow an argument specifying what variable to save the state to? =
I
> needed this myself so I wrote the following:
>=20
> #ifdef WITH_THREAD
> #  define MY_BEGIN_ALLOW_THREADS(st)    \
>     { st =3D PyEval_SaveThread(); }
> #  define MY_END_ALLOW_THREADS(st)      \
>     { PyEval_RestoreThread(st); st =3D NULL; }
> #else
> #  define MY_BEGIN_ALLOW_THREADS(st)
> #  define MY_END_ALLOW_THREADS(st)      { st =3D NULL; }
> #endif
>=20
> It works just fine but has one drawback: Whenever Py_BEGIN_ALLOW_THREAD=
S
> changes, I have to change my macros too.
>=20
> Wouldn't it be reasonable to supply two sets of macros, one that allows
> exactly this, and one that does what Py_BEGIN_ALLOW_THREADS currently
> does.

If you're wondering why I'm replying to my own age-old-mail, it is becaus=
e
I noticed the following snippet in python-announce-list:

"""
Martin Sj=F6gren wondered why the Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS macros don't take an argument specifying a variable
to save the thread state to.  He said he needed this, but didn't explain
why.
"""

Looking back I see that, no, I did not motivate that one bit :)

Here's the scoop:

One of the functions (say foo()) in my extension module calls another
function (real_foo()) which might block since it does creepy things like
I/O.  Being a nice fella I surrounded the call with Py_BEGIN_ALLOW_THREAD=
S
and Py_END_ALLOW_THREADS so that other threads could jump in and do stuff.

The problem is that real_foo() might call a previously defined callback
(written in C) which in turn has to call a Python callback (what good is
an extension module if you have to write your callbacks in C?), but when =
I
try to PyEval_CallObject() it - KABOOM!  Segfault somewhere in the
interpreter (on PyFrame_New() or something like that).

After a lot of thinking (and a lot of help from python-list) I figured ou=
t
that the state I push away using Py_BEGIN_ALLOW_THREADS must be retrieved
again before the callback is called, or PyFrame_New() won't like me.
Problem is that that state variable is declared in Py_BEGIN_ALLOW_THREADS
and there's no way to reach it (and it also includes a naughty { with the
corresponding } in the Py_END_ALLOW_THREADS so it won't work to use them
anyway).

My solution: I wrote the two macros above.  This works for me, but
obviously I have to update my macros when the "real" macros in the Python
distribution change.


Okay, that's a more thorough explanation, I hope everybody are happy :)

Martin

--=20
Martin Sj=F6gren
  martin@strakt.com              ICQ : 41245059
  Phone: +46 (0)31 405242        Cell: +46 (0)739 169191
  GPG key: http://www.strakt.com/~martin/gpg.html