[C++-sig] Managing the GIL across competing threads

[Adam Preble]

I edited the post so many times I forgot the subject line!  I am a bad
person.

Maybe the most condensed question here is: How do you normally manage
multiple resources competing with the GIL in a way that could cause
deadlocks with it?  For the details, see the long post.

On Thu, Mar 15, 2012 at 11:59 PM, Adam Preble <adam.preble at gmail.com> wrote:

> I discovered recently that during callbacks to Python objects, sometimes
> the interpreter would try to do stuff at the same time, despite the fact I
> made a call to ensure the GIL.  I read the solution for that kind of thing
> I'm doing is calling PyEval_InitThreads().  This worked at first, but like
> more race conditions and stuff, all it takes is walking away from the
> computer for it all to fall apart.
>
> What I'm seeing is a rather elaborate deadlock situation revolved around
> securing the GIL.  I think I need to basically put my interpreter in its
> own subsystem, design-wise, and ferret calls to invoke things in the
> interpreter to it, in order to ultimately get around this.  However, what
> I'm asking of the distribution is how they've gotten around this.
>
> To give something a little more concrete--this is pretty gross:
> 1. Main thread starts interpreter and is running a script
> 2. The script defines an implementation of callback interface A
> 3. The script starts some objects that represent threads in the C++
> runtime.  These are threads 1 and 2.
> 4. The script starts to create an object that is wrapped from C++
> 5. The object requires a resource from thread 1, where I use promises and
> futures enqueue and fulfill the request when thread #1 isn't busy.
> 6. Meanwhile, the interpreter thread is waiting for the resource since it
> cannot put it off any further
> 7. At this point, thread 2 tries to invoke the callback to interface A,
> and it needs the interpreter thread.
> 8. thread #1 needs thread #2 to complete this critical step before the
> get() call will complete for the master interpreter thread
> 9. Thread #2 needs thread #1 to finish so it can get the GIL.  It's
> basically locked up in PyGILState_Ensure().
>
> Heck of a deadlock.  I am pondering having something else control the
> interpreter in its own thread and have everybody enqueue stuff up to run in
> it, like with the promises and futures I'm using elsewhere already.  The
> reason is that thread #2 doesn't really need to steal the interpreter at
> that very exact moment.  And furthermore, I'm trying to use Stackless, and
> it's my understanding there I can link it up so that the interpreter gets
> ahold of the Python runtime at controlled intervals--if desired--to invoke
> other stuff.
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120316/258454e5/attachment-0001.html>