[Python-Dev] Extension modules, Threading, and the GIL

[Martin v. Löwis]

David Abrahams <dave@boost-consulting.com> writes:

> > Wrong. If the code in B that calls Q does not allow threads, the
> > callbacks don't need to reacquire the GIL.
> 
> I think you must be misunderstanding me.  These callbacks might be
> invoked on threads that are not the Python main thread.  

Which thread is the main thread is not relevant at all.

What matters is whether the callback is invoked in the context of the
thread that installed it (i.e. as a result of calling a function in B).

>    "the rule exists that only the thread that has acquired the global
>    interpreter lock may operate on Python objects or call Python/C API
>    functions"
> 
> I am taking that statement at face value and assuming it means what it
> says.

And rightly so. Notice that it does not use the term "main thread".

> > Can you please explain what a callback is? 
> 
> I'm trying to leave out irrelevant details, but a callback happens to
> be a virtual function in a C++ class instance in this case.  These
> callbacks implement behaviors of base classes supplied by the library,
> Qt.  

Ok, now I know how a callback is installed (by redefining a virtual
method). The other question is: How is it invoked? I.e. who invokes
it, and why?

I suppose the immediate answer is "the library Q". However, that
library does not invoke it with out a trigger: What is that trigger?

> What matters, AFAICT, is that the callback might be invoked on a
> thread that's not Python's main thread, thus must acquire the GIL,
> so if Python's main thread wants to call something in Q that might
> invoke one of these callbacks, it must release the GIL.

Which thread is the main thread is completely irrelevant. If you have
something like

class MyB(B.Base):
  def overridden(self):
    print "This overrides a virtual function"

def doit():
  b = MyB()
  B.do_the_real_work(b) # will call Q, which will call the callback,
                        # which will call overridden

then there is no need for the C++ code in B to release the GIL, nor is
there a need for B to reacquire the GIL in the callback. This is
independent of whether doit is called in the main thread, or in the
context of any other thread: at the point where do_the_real_work is
called, the current thread holds the GIL. If it keeps the GIL, then
the GIL will be held at the point when the callback occurs, in the
context of the thread where the callback occurs.

> Maybe we have different ideas of what "callback" means.  In your
> terms, it is not a "true callback".

Then I'm still curious as to what triggers invocation of that virtual
function.

Regards,
Martin