[C++-sig] (no subject)

[Adam Preble]

I discovered recently that during callbacks to Python objects, sometimes
the interpreter would try to do stuff at the same time, despite the fact I
made a call to ensure the GIL.  I read the solution for that kind of thing
I'm doing is calling PyEval_InitThreads().  This worked at first, but like
more race conditions and stuff, all it takes is walking away from the
computer for it all to fall apart.

What I'm seeing is a rather elaborate deadlock situation revolved around
securing the GIL.  I think I need to basically put my interpreter in its
own subsystem, design-wise, and ferret calls to invoke things in the
interpreter to it, in order to ultimately get around this.  However, what
I'm asking of the distribution is how they've gotten around this.

To give something a little more concrete--this is pretty gross:
1. Main thread starts interpreter and is running a script
2. The script defines an implementation of callback interface A
3. The script starts some objects that represent threads in the C++
runtime.  These are threads 1 and 2.
4. The script starts to create an object that is wrapped from C++
5. The object requires a resource from thread 1, where I use promises and
futures enqueue and fulfill the request when thread #1 isn't busy.
6. Meanwhile, the interpreter thread is waiting for the resource since it
cannot put it off any further
7. At this point, thread 2 tries to invoke the callback to interface A, and
it needs the interpreter thread.
8. thread #1 needs thread #2 to complete this critical step before the
get() call will complete for the master interpreter thread
9. Thread #2 needs thread #1 to finish so it can get the GIL.  It's
basically locked up in PyGILState_Ensure().

Heck of a deadlock.  I am pondering having something else control the
interpreter in its own thread and have everybody enqueue stuff up to run in
it, like with the promises and futures I'm using elsewhere already.  The
reason is that thread #2 doesn't really need to steal the interpreter at
that very exact moment.  And furthermore, I'm trying to use Stackless, and
it's my understanding there I can link it up so that the interpreter gets
ahold of the Python runtime at controlled intervals--if desired--to invoke
other stuff.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120315/522c17f7/attachment.html>