[Async-sig] killing tasks that won't cancel

[Nathaniel Smith]

I'm not sure what a "higher priority" exception is...raising an
exception is hard to miss.

There are a few things Trio does differently here that might be
relevant, but it depends on why Chris is having trouble cancelling
tasks.

1. Trio's cancellation exception, trio.Cancelled, inherits from
BaseException instead of Exception, like KeyboardInterrupt or
StopIteration. So 'except Exception' doesn't catch it by accident.

2. Trio's cancellation is "stateful": if your code is in the cancelled
state, then every time you try to do an async operation then it raises
trio.Cancelled again. So you avoid the case where a tasks gets stuck,
someone forces it to raise CancelledError, and then it has a 'finally'
block that tries to do some cleanup... but the 'finally' block also
gets stuck. In trio the 'finally' block can't accidentally get stuck.

3. Both of these features are somewhat dependent on trio using
"delimited" cancellation. Before you can cancel something, you have to
say how far you want to unwind. This means that there's never any
reason for anyone to try to catch 'Cancelled' on purpose, because trio
will catch it for you at the appropriate moment. And it's hard to do
'stateful' cancellation if you don't know how long the state is
supposed to persist. And, you avoid cases where some code thinks it
just threw in a CancelledError and is supposed to catch it, but
actually it was thrown in from some other stack frame, and it ends up
confusedly catching the wrong exception.

I'm not sure how much of this could be adapted for asyncio. The
obvious change would be to make asyncio.CancelledError a
BaseException, though it seems borderline to me from a back-compat
perspective. I think I remember Yury was thinking about changing it
anyway, though? That would definitely help with the 'except Exception'
kind of mistake.

But the other issues are deeper. If you don't have a solid system for
keeping track of what exactly is supposed to be cancelled, then it's
easy to accidentally cancel too much, or cancel too little. Solving
that requires a systematic approach. And unfortunately, asyncio
already has 2 different sets of cancellation semantics (Future.cancel
-> takes effect immediately, irrevocable & idempotent, doesn't
necessarily cause the underlying machinery to stop processing, just
stops it from reporting its result; Task.cancel -> doesn't take effect
immediately or necessarily at all, can be called repeatedly and
injects one CancelledError per call, tries to stop the underlying
machinery, chains to other tasks/futures that the first task is
await'ing). So if our goal is to make the system as a whole as
reliable and predictable as possible within the constraints of
back-compat... I don't know whether adding a third set of semantics
would actually help, or make more code confused about what it was
supposed to be catching.

And I don't know if any of these actually address whatever problem
you're having with uncancellable tasks. It's certainly possible to
make an uncancellable task in Trio too. We just try to make it hard to
do by accident.

-n

On Tue, Feb 19, 2019 at 3:53 PM Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>
> On Tue, Feb 19, 2019 at 12:33 PM Yury Selivanov <yselivanov at gmail.com> wrote:
>>
>> Unfortunately asyncio isn't super flexible around "cancellation with a timeout" kind of scenarios. The current assumption is that once the cancellation is requested, the Task will start cancelling and will do so in a timely manner.  Imposing a second layer of timeouts on the cancellation process itself isn't natively supported.  But to properly address this we don't need a very broadly defined Task.set_exception(); we need to rethink the cancellation in asyncio (perhaps draw some inspiration from Trio and other frameworks).
>
>
> What options have you already considered for asyncio's API? A couple naive things that occur to me are adding task.kill() with higher priority than task.cancel() (or equivalently task.graceful_cancel() with lower priority). Was task.cancel() meant more to have the meaning of "kill" or "graceful cancel"? In addition, the graceful version of the two (whichever that may be) could accept a timeout argument -- after which the exception of higher priority is raised. I realize this is a more simplistic model compared to the options trio is considering, but asyncio has already gone down the path of the simpler approach.
>
> --Chris
>
>
>>
>>
>> Yury
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
Nathaniel J. Smith -- https://vorpus.org