[issue40487] Unexpected exception handler behavior in Jupyter when returning task objects created with create_task

[Kyle Stanley]

Kyle Stanley <aeros167 at gmail.com> added the comment:

John Smith wrote:
> The code works normally again if I remove the return statement from the `run_coro` function.

I am a bit surprised that the code works as intended simply by removing the return statement. Perhaps Yury or Andrew can clarify on that point.

>From looking at the code above, the main issue seems to be that the tasks are not awaited or cancelled and cleaned up at any point. When creating a task without ever awaiting it, there's no guarantee that it will be completed within the duration of the event loop. This is handled for the user when using asyncio.run() by cancelling the tasks and propagating any exceptions during event loop finalization, but should be done manually or by the event loop when not using it. For an example, see _cancel_all_tasks() in https://github.com/python/cpython/blob/d699d5e6178adca785a8701c32daf5e18fad0bf1/Lib/asyncio/runners.py#L54.

So, my best guess as to what's happening is that the tasks are still in progress while the event loop is finalizing, and the exception handler isn't called. But, I can't say that for sure without knowing the implementation of the Jupyter event loop.

For now, my recommended solution would be to include something like the above _cancel_all_tasks() or simply await your tasks at the end to ensure they are completed before the event loop is finalized. Alternatively, Jupyter could include something like that in their event loop finalization process, so that users don't have to include it on their own when using asyncio.

I would also consider using asyncio.run(), but I'm not certain if it works correctly in a Jupyter notebook. I'm aware that it's not always a viable option when it is desired to use an existing event loop instead of creating a separate one; that's why I'm not explicitly recommending it for this situation.

----------
nosy: +aeros

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40487>
_______________________________________