[New-bugs-announce] [issue26333] Multiprocessing imap hangs when generator input errors

[Aaron Halfaker]

New submission from Aaron Halfaker:

multiprocessing.imap will hang and not raise an error if an error occurs in the generator that is being mapped over.  I'd expect the error to be raised and/or the process to fail.  

For example, run the following code in python 2.7 or 3.4:

    from multiprocessing import Pool

    def add_one(v):
        return v+1

    pool = Pool(processes=2)

    values = ["1", "2", "3", "4", "foo", "5", "6", "7", "8"]
    value_iter = (int(v) for v in values)

    for new_val in pool.imap(add_one, value_iter):
        print(new_val)

And output should look something like this:

    $ python demo_hanging.py 
    2
    3
    4
    5
    Exception in thread Thread-2:
    Traceback (most recent call last):
      File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
        self.run()
      File "/usr/lib/python3.4/threading.py", line 868, in run
        self._target(*self._args, **self._kwargs)
      File "/usr/lib/python3.4/multiprocessing/pool.py", line 378, in _handle_tasks
        for i, task in enumerate(taskseq):
      File "/usr/lib/python3.4/multiprocessing/pool.py", line 286, in <genexpr>
        self._taskqueue.put((((result._job, i, func, (x,), {})
      File "demo_hanging.py", line 9, in <genexpr>
        value_iter = (int(v) for v in values)
    ValueError: invalid literal for int() with base 10: 'foo'

The script will then hang indefinitely.

----------
components: Library (Lib)
messages: 260032
nosy: Aaron Halfaker
priority: normal
severity: normal
status: open
title: Multiprocessing imap hangs when generator input errors
type: behavior
versions: Python 2.7, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26333>
_______________________________________