[issue38084] multiprocessing cannot recover from crashed worker

[STINNER Victor]

STINNER Victor <vstinner at python.org> added the comment:

I converted the example into attached file mp_exit.py and I added a call to faulthandler to see what is going on.

Output with the master branch of Python:

vstinner at apu$ ~/python/master/python ~/mp_exit.py 
Timeout (0:00:05)!
Thread 0x00007ff40139a700 (most recent call first):
  File "/home/vstinner/python/master/Lib/multiprocessing/connection.py", line 379 in _recv
  File "/home/vstinner/python/master/Lib/multiprocessing/connection.py", line 414 in _recv_bytes
  File "/home/vstinner/python/master/Lib/multiprocessing/connection.py", line 250 in recv
  File "/home/vstinner/python/master/Lib/multiprocessing/pool.py", line 576 in _handle_results
  File "/home/vstinner/python/master/Lib/threading.py", line 882 in run
  File "/home/vstinner/python/master/Lib/threading.py", line 944 in _bootstrap_inner
  File "/home/vstinner/python/master/Lib/threading.py", line 902 in _bootstrap

Thread 0x00007ff401b9b700 (most recent call first):
  File "/home/vstinner/python/master/Lib/multiprocessing/pool.py", line 528 in _handle_tasks
  File "/home/vstinner/python/master/Lib/threading.py", line 882 in run
  File "/home/vstinner/python/master/Lib/threading.py", line 944 in _bootstrap_inner
  File "/home/vstinner/python/master/Lib/threading.py", line 902 in _bootstrap

Thread 0x00007ff40239c700 (most recent call first):
  File "/home/vstinner/python/master/Lib/selectors.py", line 415 in select
  File "/home/vstinner/python/master/Lib/multiprocessing/connection.py", line 930 in wait
  File "/home/vstinner/python/master/Lib/multiprocessing/pool.py", line 499 in _wait_for_updates
  File "/home/vstinner/python/master/Lib/multiprocessing/pool.py", line 519 in _handle_workers
  File "/home/vstinner/python/master/Lib/threading.py", line 882 in run
  File "/home/vstinner/python/master/Lib/threading.py", line 944 in _bootstrap_inner
  File "/home/vstinner/python/master/Lib/threading.py", line 902 in _bootstrap

Thread 0x00007ff4102cf740 (most recent call first):
  File "/home/vstinner/python/master/Lib/threading.py", line 303 in wait
  File "/home/vstinner/python/master/Lib/threading.py", line 565 in wait
  File "/home/vstinner/python/master/Lib/multiprocessing/pool.py", line 759 in wait
  File "/home/vstinner/python/master/Lib/multiprocessing/pool.py", line 762 in get
  File "/home/vstinner/python/master/Lib/multiprocessing/pool.py", line 364 in map
  File "/home/vstinner/mp_exit.py", line 12 in <module>


In the main process, Pool._handle_results() thread is blocked on os.read() which never completes, even if the child process died and so the other end of the pipe should be closed.

----------
Added file: https://bugs.python.org/file48603/mp_exit.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38084>
_______________________________________