[Python-Dev] Extension modules, Threading, and the GIL

[David Abrahams]

"Martin v. L=F6wis" <martin@v.loewis.de> writes:

> David Abrahams wrote:
>> Python extension module B calls shared library Q and uses Q's callback
>> interface.  Because some of the callbacks need to use the Python API,
>> and *might* be invoked by threads, they must all acquire the GIL.
>
> Wrong. If the code in B that calls Q does not allow threads, the
> callbacks don't need to reacquire the GIL.

I think you must be misunderstanding me.  These callbacks might be
invoked on threads that are not the Python main thread.=20=20

http://www.python.org/doc/current/api/threads.html says:

   "the rule exists that only the thread that has acquired the global
   interpreter lock may operate on Python objects or call Python/C API
   functions"

I am taking that statement at face value and assuming it means what it
says.  Note that these threads were not started by Python's thread API.

>> Problem: using B while A is loaded breaks A: because B has installed
>> callbacks in Q that acquire the GIL, A must also release the GIL
>> before calling into Q.
>
> Can you please explain what a callback is?=20

I'm trying to leave out irrelevant details, but a callback happens to
be a virtual function in a C++ class instance in this case.  These
callbacks implement behaviors of base classes supplied by the library,
Qt.  For the purposes of this discussion, it might just as well be a
'C' language pointer-to-function, though:

           void (*)()

> Can the callbacks occur in
> the context of a different thread, i.e. different from the one that
> has installed the callback?

I think the answer is yes, but I'm trying to leave out the irrelevant,
and I'm pretty sure that it doesn't matter one whit which thread
installs the callback.  What matters, AFAICT, is that the callback
might be invoked on a thread that's not Python's main thread, thus
must acquire the GIL, so if Python's main thread wants to call
something in Q that might invoke one of these callbacks, it must
release the GIL.

> If it is a true callback (i.e. Q will call B back while being called
> from B), then this won't interfere at all with A.

Maybe we have different ideas of what "callback" means.  In your
terms, it is not a "true callback".

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution