[New-bugs-announce] [issue28382] Possible deadlock after many multiprocessing.Process are launch

[Alexis]

New submission from Alexis:

I am launching a process inside a pool worker, using the multiprocessing module.
After a while, a deadlock append when I am trying to join the process.

Here is a simple version of the code:

import sys, time, multiprocessing
from multiprocessing.pool import ThreadPool

def main():
    # Launch 8 workers
    pool = ThreadPool(8)
    it = pool.imap(run, range(500))
    while True:
        try:
            it.next()
        except StopIteration:
            break

def run(value):
    # Each worker launch its own Process
    process = multiprocessing.Process(target=run_and_might_segfault,     args=(value,))
    process.start()

    while process.is_alive():
        sys.stdout.write('.')
        sys.stdout.flush()
        time.sleep(0.1)

    # Will never join after a while, because of a mystery deadlock
    process.join()

def run_and_might_segfault(value):
    print(value)

if __name__ == '__main__':
    main()

And here is a possible output:

~ python m.py
..0
1
........8
.9
.......10
......11
........12
13
........14
........16
........................................................................................
As you can see, process.is_alive() is alway true after few iterations, the process will never join.

If I CTRL-C the script a get this stacktrace:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 680, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "m.py", line 30, in <module>
    main()
  File "m.py", line 9, in main
    it.next()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5    /lib/python3.5/multiprocessing/pool.py", line 684, in next
    self._cond.wait(timeout)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5    /lib/python3.5/threading.py", line 293, in wait
    waiter.acquire()
KeyboardInterrupt

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5    /lib/python3.5/multiprocessing/popen_fork.py", line 29, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

Using python 3.5.1 on macos, also tried with 3.5.2 with same issue.
Same result on Debian.
I tried using python 2.7, and it is working well. May be a python 3.5 issue only?

Here is the link of the stackoverflow question:
http://stackoverflow.com/questions/39884898/large-amount-of-multiprocessing-process-causing-deadlock

----------
components: Library (Lib)
messages: 278221
nosy: Hadhoke
priority: normal
severity: normal
status: open
title: Possible deadlock after many multiprocessing.Process are launch
type: behavior
versions: Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28382>
_______________________________________