[New-bugs-announce] [issue33469] RuntimeError after closing loop that used run_in_executor

[Hrvoje Nikšić]

New submission from Hrvoje Nikšić <hniksic at gmail.com>:

Looking at a StackOverflow question[1], I was unable to find a way to correctly close an event loop that uses run_in_executor() for long-running tasks.

The question author tried to implement the following scenario:

1. create some tasks that use run_in_executor
2. run asyncio.wait(tasks, return_when=FIRST_EXCEPTION)
3. cancel pending tasks, if any
4. close the loop and continue with non-async work

However, when there are pending tasks from wait(), a RuntimeError is raised some time after step #4. In the test programs, it happens while waiting for the program to finish. I have attached a minimal example to reproduce the issue.

The immediate cause is that a callback installed by wrap_future() notices that the underlying concurrent.futures.Future is done and calls loop.call_soon_threadsafe() to copy the result to the asyncio.Future. call_soon_threadsafe() fails when the loop is closed.

This would be reasonable behavior if not for the fact that the code explicitly cancelled the asyncio future, and awaited it to ensure that the cancellation took effect. While it is clear that asyncio cannot interrupt a function already running in an executor, it should probably detach the connection between the concurrent future and the asyncio future, to prevent this kind of error (and possibly other problems).

For example, the _call_check_cancel closure in _chain_future could remove (or disable) the done_callback installed on source after the call to source.cancel(). Since at that point we know that destination (asyncio.Future) is already canceled, there is no longer value in invoking the done callback for source (concurrent.futures.Future).


[1]
https://stackoverflow.com/q/50279522/1600898

----------
files: executor-cancel
messages: 316420
nosy: hniksic
priority: normal
severity: normal
status: open
title: RuntimeError after closing loop that used run_in_executor
versions: Python 3.6, Python 3.7
Added file: https://bugs.python.org/file47583/executor-cancel

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33469>
_______________________________________