[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Hrvoje Nikšić]

New submission from Hrvoje Nikšić <hniksic at gmail.com>:

At interpreter shutdown, Python waits for all pending futures of all executors to finish. There seems to be no way to disable the wait for pools that have been explicitly shut down with pool.shutdown(wait=False). The attached script demonstrates the issue.

In our code the futures are running blocking network calls that can be canceled by the user. The cancel action automatically cancels the pending futures and informs the running ones that they should exit. The remaining futures are those whose callables are "stuck" in network calls with long or infinite timeouts, such as reading from a non-responding network filesystem. Since those can't be interrupted, we use pool.shutdown(wait=False) to disown the whole pool. This works nicely, until the application exit, at which point it blocks trying to wait for the pending futures to finish. This can take an arbitrary amount of time, possibly never finishing.

A similar question has come up on StackOverflow, with the only answer recommending to unregister concurrent.futures.thread._python_exit: https://stackoverflow.com/q/48350257/1600898 . We are ourselves using a similar hack, but we would like to contribute a proper way to disown an executor.

The current behavior is explicitly documented, so presumably it can't be (easily) changed, but we could add a "disown" keyword argument to shutdown(), or a new disown() method, which would serve to explicitly disable the waiting. If this is considered desirable, I will create a pull request.

----------
components: Library (Lib)
files: pool-shutdown
messages: 341329
nosy: hniksic
priority: normal
severity: normal
status: open
title: Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors
versions: Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file48297/pool-shutdown

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36780>
_______________________________________