[issue43389] Cancellation ignored by asyncio.wait_for can hang application

[nmatravolgyi]

New submission from nmatravolgyi <nandor.matra at gmail.com>:

I have found myself debugging a *very* not intuitive behavior regarding asyncio.wait_for that I'd consider a bug/deficiency. The problem very simply put: wait_for will return the wrapped future's result even when it is being cancelled, ignoring the cancellation as it has never existed.

This will make parallel execution-waits hang forever if some simple conditions are met. From the perspective of this snippet every task must exit so it just needs to wait. I know cancellation *can* be ignored, but it is discouraged by the documentation for this reason exactly.

tasks = [...]
for t in tasks:
    t.cancel()
results = await asyncio.gather(*tasks, return_exceptions=True)

I already know that this behavior has been chosen because otherwise the returned value would be lost. But for many applications, losing an explicit cancellation error/event is just as bad.

The reason why ignoring the cancellation is critical is because the cancelling (arbiter) task cannot reliably solve it. In most cases having repeated cancellations in a polling wait can solve this, but it is ugly and does not work if the original wait_for construct is in a loop and will always ignore the cancellation.

The most sensible solution would be to allow the user to handle both the return value and the cancellation if they do happen at once. This can be done by subclassing the CancelledError as CancelledWithResultError and raising that instead. If the user code does not handle that exception specifically then the user "chose" to ignore the result. Even if this is not intuitive, it would give the user the control over what really is happening. Right now, the user cannot prefer to handle the cancellation or both.

Lastly, I may have overlooked something trivial to make this work well. Right now I'm considering replacing all of the asyncio.wait_for constructs with asyncio.wait constructs. I can fully control all tasks and cancellations with that. I've made a simple demonstration of my problem, maybe someone can shed some light onto it.

----------
components: asyncio
files: aio_wait_for_me.py
messages: 388032
nosy: asvetlov, nmatravolgyi, yselivanov
priority: normal
severity: normal
status: open
title: Cancellation ignored by asyncio.wait_for can hang application
type: behavior
versions: Python 3.8, Python 3.9
Added file: https://bugs.python.org/file49849/aio_wait_for_me.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43389>
_______________________________________