reentrant callbacks from C into Python

[Thomas Heller]

gb at cs.unc.edu writes:

> I'm developing a wrapper for the Microsoft ActiveAccessibility, and
> SetWinEventHook API's. Most things are working fine but occasionally I
> get a reentrant callback on a WinEvent. They advise that this is
> possible and say users should guard against it but don't say how.
> 
> Here's how it happens:
> 
> On SetWinEventHook, I squirrel away a pointer to the Python callable
> (call it "phook") that is provided for the callback. The C function
> SetWinEventHook is called with a C callback function "chook".
> 
> When chook is called from somewhere outside of python, it first
> acquires the GIL, the calls phook (back into python), when the
> phook returns, chook releases the GIL.
> 
> Problem is that some of the functions in the API can cause the
> callback to be reentered. So, I'm processing a callback and one of the
> calls in the Python code causes the C callback to be reentered.
> 
> Now when I try to acquire the GIL, deadlock ensues. 
> 
> I've made a hack workaround with a static variable keeping track of
> the number of times I've locally acquired the GIL. If I already have
> it I don't try to get it again. That appears to work but I wonder if
> it would work in the presence of threads.
> 
> I thought about releasing the GIL around the API functions that I am
> wrapping but they warn the reentry might happen in lots of functions
> outside the ActiveAccessibility and WinEvents APIs. So it is hard to
> be sure that the GIL won't already be held when I enter the C
> callback.

The usual (?) way is to release the GIL on every call to a C function
which may trigger callbacks into Python, and acquire a new thread state
on each callback calling into Python again.  At least that's how ctypes
does it, and I believe win32all uses a somewhat similar scheme.

> 
> There *must* be a better way. Is there a standard way to deal with a
> problem like this?
> 

The above sounds expensive, but works very well in practice.

Thomas