[issue10915] Make the PyGILState API compatible with multiple interpreters

[Graham Dumpleton]

Graham Dumpleton <Graham.Dumpleton at gmail.com> added the comment:

The bulk of use cases is going to be simple callbacks via the same thread that called out of Python in the first place. Thus ultimately all it is doing is:

Py_BEGIN_ALLOW_THREADS

Call into some foreign C library.
C library wants to do a callback into Python.

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

More stuff in C library.
Return back into the C extension wrapper.

Py_END_ALLOW_THREADS

This is what SWIG effectively does in its generated wrappers for callbacks.

Using a TLS solution, all these modules that simply do this will now start working where as they currently usually deadlock or have other problems.

In your solution, all these modules would need to be modified to some how transfer information about the current interpreter into the callback which is called by the foreign C library and use new PyGILState_??? functions rather than the old.

I do accept that more complicated extension modules which create their own foreign threads and perform the call back into interpreter from that thread, or systems like mod_wsgi which have a persistent thread pool from which calls originate, will have to be modified, but this is the lessor use case from what I have seen.

Overall, it is an easy win if TLS is used because a lot of code wouldn't need to change. Some will, but expect that a lot of the common stuff like lxml for example wouldn't.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10915>
_______________________________________