[Python-Dev] Questions about signal handling.

[Eric Snow]

On Tue, Sep 25, 2018 at 1:45 AM Victor Stinner <vstinner at redhat.com> wrote:
> Please don't rely on this ugly API. *By design*, Py_AddPendingCall()
> tries 100 times to acquire the lock: if it fails to acquire the lock,
> it does notthing... your callback is ignored...

Yeah, there are issues with pending calls as implemented.
Furthermore, I'm not clear on why it was made a public API in the
first place.  Ultimately I'd like to deprecate Py_AddPendingCall and
Py_MakePendingCalls (but that's not my priority right now).
Regardless, the underlying machinery matches what I need for
interpreter isolation right now.  See below.

> By the way, recently, we had to fix yet another bug in signal
> handling. A new function has been added:
> [snip]

I saw that.  If anything, it's more justification for separating
signals from the pending calls machinery. :)

> If you want to exchange commands between two interpreters, which I see
> as two threads, I suggest to use two queues and something to consume
> queues.

Sure.  However, to make that work I'd end up with something that's
almost identical to the existing pending calls machinery.  Since the
function-to-run may call Python code, there must be an active
PyThreadState and the GIL (or, eventually, target interpreter's lock)
must be held.  That is what Py_MakePendingCalls gives us already.
Injecting the pending call into the eval loop (as we do today) means
we don't have to create a new thread just for handling pending calls.
[1]

In the short term I'd like to stick with existing functionality as
much as possible.  Doing so has a number of benefits.  That's been one
of my guiding principles as I've worked toward the multi-core Python
goal.  That said, I agree that the pending calls machinery has some
deficiencies that should be fixed or something better should replace
it.  I just don't want that to get in the way of short-term goals,
especially as I have limited time for this.

-eric

[1] FWIW, a separate thread for "asynchronous" operations (ones that
interrupt the eval loop, e.g. "pending" calls and signals) might be a
viable approach now that we require platforms to support threading.
The way we interrupt the eval loop currently seems more complex (and
inefficient) than necessary.  I was thinking about this last week and
plan to explore it further at some point in the future.  For now,
though, I'd like to keep the focus on more immediate needs. :)