[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

[Carlo Rosati]

Carlo Rosati <crosati777 at icloud.com> added the comment:

I figured out that the problem is itertools.tee does not use a multiprocessing.Manager proxied object for shared state. I was able to create a workaround tee as follows.

def multiprocessing_tee(iterable, n=2):
    """Write a multiprocessing safe itertools.tee"""
    it = iter(iterable)
    m = multiprocessing.Manager()
    lists = [m.list() for i in range(n)]
    def gen(local_list):
        keep_m_alive = m
        while True:
            if not local_list:         # when the local list is empty
                newval = next(it)      # fetch a new value and
                for l in lists:        # load it to all the lists
                    l.append(newval)
            yield local_list.pop(-1)
    return tuple(gen(l) for l in lists)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34410>
_______________________________________