[issue1497532] C API to retain GIL during Python Callback

[Lukas Lueg]

Lukas Lueg <knabberknusperhaus at yahoo.de> added the comment:

I'm not sure if such a API is feasible. The very nature of Python makes
it impossible to tell in advance what the interpreter will do when
getting called. This is even true for simple functions - think of your
function getting decorated...

Let's consider the following scenario:

- Python-Thread 1 is already running and owns a lock on some object.
While it still owns the lock, it releases the GIL.
- We are executing in Python-Thread 2 and call from Python to C. The C
function has the GIL, "locks" it and calls back to Python.
- The Python function executes in the same thread, still having the GIL.
It tries to acquire the lock on the same object as Thread 1. Preventing
a deadlock between those two threads, it releases the GIL and waits for
the object-lock.
- The GIL is "locked" to the current thread and the current thread is
the only one that we can allow to unlock it again; this narrows our
options down to Py_BEGIN_ALLOW_THREADS becoming a No-Op in such situations.
- Py_BEGIN_ALLOW_THREADS executed in the second C-function is silently
ignored. Thread 2 waits for the object-lock with Thread 1 never having a
chance to release it.
- The interpreter just deadlocked.


AFAICS we can't guarantee not to deadlock if there are other threads
running before we lock the GIL to the current thread.

----------
nosy: +ebfe

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1497532>
_______________________________________