[issue39360] python3.8 regression - ThreadPool join via __del__ hangs forever

[Pablo Galindo Salgado]

Pablo Galindo Salgado <pablogsal at gmail.com> added the comment:

This is not a regression: the code only worked on 3.7 because there is a sleep in the pool maintainance loop that makes the pool more likely being more consistent, but is the same problem as there is a race condition. For example, reducing the sleep in the loop (notice 3.8 does not use a sleep but instead uses a more resilient system of sentinels) for the 3.7 version reproduces the problem:

diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py
index 3e9a0d6b48..f8d438d87c 100644
--- a/Lib/multiprocessing/pool.py
+++ b/Lib/multiprocessing/pool.py
@@ -410,7 +410,7 @@ class Pool(object):
         # is terminated.
         while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
             pool._maintain_pool()
-            time.sleep(0.1)
+            time.sleep(0.00000001)
         # send sentinel to stop workers
         pool._taskqueue.put(None)
         util.debug('worker handler exiting')

With that patch, 3.7 hangs as well. The problem here is that something regarding the locks inside the SimpleQueue of inbound tasks is not in a consistent state when Python finalization calls __del__.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39360>
_______________________________________