[Python-ideas] Responsive signal handling

[Cameron Simpson]

On 22Jun2015 01:42, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
>On Mon, Jun 22, 2015 at 12:52 AM, Cameron Simpson <cs at zip.com.au> wrote:
>> On 21Jun2015 19:16, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
>>> Operations that run in C but just
>>> take a long time, or that are part of third-party code, will continue
>>> to inhibit responsiveness.
>>>
>>> - Run signal handlers in a dedicated separate thread.
>>>
>>> IMO this is generally better than running signal handlers in the main
>>> thread, because it eliminates the separate concept of "async-safe" and
>>> just requires "thread-safe". So you can use regular threading
>>> synchronization primitives for safety, instead of relying on luck /
>>> memorized lists of atomic/re-entrant operations.
>>
>> Yes, I am in favour of this or something like it. Personally I would go for
>> either or both of:
>>
>>  - a stdlib function to specify the thread to handle signals instead of main
>
>This just moves the problem to another thread. One can already today
>try to keep the main thread free to handle signals, it's just hard.

Yes. But it is very easy to ensure that a specifial purpose Thread is free to 
handle signals. And it is arguably the minimalist change.

>>  - a stdlib function to declare that signals should immediately place a nice
>> descriptive "signal" object on a Queue, and leaves it to the user to handle
>> the queue (for example, by spawning a thread to consume it)
>
>I like this. It mirror's Linux's selectfd, too. One small correction,
>it can't literally be a Queue, because those aren't safe to use in
>signal handlers. (It can be a pipe that is wrapped in a Queue-like
>interface, though, and if we do that, we can even use native signalfd
>if we want.)
>
>It also resolves an unspoken concern I had, which is that silently
>starting threads for the user feels icky.

I wasn't proposing silently starting threads. I imagined the former suggestion 
would be handed a thread as the signal target.

>>> Something still needs to run in the main thread though, for e.g.
>>> KeyboardInterrupt, so this is not super straightforward.
>>
>> Is this necessarily true?
>
>What I mean is that there needs to be a way to raise KeyboardInterrupt
>in the main thread from a signal handler. If, as you suggest, the old
>behavior stays around, then that's enough.

I was imagining the old behaviour stayed around by default, not necessarily as 
fixed behaviour. But "KeyboardInterrupt occurs in the main thread" is handy.

Perhaps a better solution here is not to keep KeyboardInterrupt special (i.e.  
always going to the main thread) but to extend "raise" to accept a thread 
argument:

  raise blah in thread

Given that signals are already presented as occuring between Python opcodes, it 
seems reasonable to me that the signal situation could be addressed with a 
common mechanism extended to exceptions.

How often is the question "how do I terminate another thread?" raised on 
python-list? Often. The standard answer is "set a flag and have the thread 
consult it". That is very sensitive to how often the flag is polled: too often 
and it infuses the code with noise (not to mention ungainly loop termination 
logic etc), too infrequently and the response is much like your issue here with 
signals: it can be arbitrarily delayed.

Suppose one could raise signals in another thread? Then the answer becomes 
"raise exception in other_thread". And the other thread will abort as soon as 
the next python opcode would fire.

It has several advantages:

  it removes any need to poll some shared state, or the set up shared state

  it lets the target thread remain nice and pythonic, letting unhandled 
  exceptions simply abort the thread automatically as they would anyway

  it lets the target thread catch the exception and handle it if desired

  it dovetails neatly with our hypothetical special signal handling thread: the 
  handling thread has merely to "raise KeyboardInterrupt in main_thread" to get 
  the behaviour you seek to preserve, _without_ making SIGINT specially handled 
  - the specialness is not an aspect of the handling thread's code, not 
  hardwired

>Another option, if we went with a dedicated signal handling thread,
>would be that uncaught exceptions propagate to the main thread when it
>gets around to it.

Perhaps. But I'd rather not; you _can_ always catch every exception and if we 
have "raise exception in thread" we can implement the above trivially for 
programs which want it.

Cheers,
Cameron Simpson <cs at zip.com.au>

Nothing is impossible for the man who doesn't have to do it.