[issue10332] Multiprocessing maxtasksperchild results in hang

[Charles-Francois Natali]

Charles-Francois Natali <neologix at free.fr> added the comment:

This problem arises because the pool's close method is called before all the tasks have completed. Putting a sleep(1) before pool.close() won't exhibit this lockup.
The root cause is that close makes the workers handler thread exit: since the maxtasksperchild argument is used, workers exit when they've processed their max number of tasks. But since the workers handler thread exited, it doesn't maintain the pool of workers anymore, and thus the remaining tasks are not treated anymore, and the task handler thread waits indefinitely (since it waits until the cache is empty).
The solution is to prevent the worker handler thread from exiting until the cache has been drained (unless the pool is terminated in which case it must exit right away).
Attached is a patch and relevant test.

Note: I noticed that there are some thread-unsafe operations (the cache that can be modified from different threads, and thread states are modified also from different threads). While this isn't an issue with the current cPython implementation (GIL), I wonder if this should be fixed.

----------
keywords: +patch
nosy: +neologix
Added file: http://bugs.python.org/file21644/pool_lifetime_close.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10332>
_______________________________________