[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Hrvoje Nikšić]

Hrvoje Nikšić <hniksic at gmail.com> added the comment:

I agree with the last comment. The disowning functionality is only used in specific circumstances, so it's perfectly fine to keep the functionality as a shutdown flag. I also agree that the change cannot be *unconditional*, for backward compatibility if not for other reasons.

The StackOverflow question, and more importantly the existence of shutdown(wait=False), suggest that there are legitimate cases when one doesn't want to wait for all running futures to finish. If a flag to shutdown() is considered to complicate the API, then perhaps we could add an opt-out by subclassing the executor and overriding a semi-private method.


Currently there seems to be no way to just abandon the thread pool. Since user threads don't and never will support cancellation, the options are:

1. add the disown option to shutdown, as suggested
2. monkey-patch concurrent.futures to not block at shutdown
3. make functions executed by the pool externally cancellable
4. roll our own thread pool

#1 is suggested by this issue, and #2 is what we do now, but it's obviously unacceptable in the long term.

#3 is infeasible because the functions we submit to the pool heavily rely on http (through "requests") and database communication, which don't support user-driven cancellation.

#4 would be technically possible, but it would require reimplementing half of concurrent.futures (badly), just for the purpose of being able to get rid of the mandatory wait at interpreter exit.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36780>
_______________________________________