Proper way to pass Queue to process when using multiprocessing.imap()?

[Chris Angelico]

On Thu, Sep 5, 2019 at 2:07 AM Israel Brewster <ijbrewster at alaska.edu> wrote:
>
> >
> > On Sep 3, 2019, at 11:09 AM, Israel Brewster <ijbrewster at alaska.edu> wrote:
> >
> >>
> >> On Sep 3, 2019, at 10:49 AM, Peter Otten <__peter__ at web.de> wrote:
> >>
> >> Israel Brewster wrote:
> >>
> >>> When using pool.imap to apply a function over a list of values, what is
> >>> the proper way to pass additional arguments to the function, specifically
> >>> in my case a Queue that the process can use to communicate back to the
> >>> main thread (for the purpose of reporting progress)? I have seen
> >>> suggestions of using starmap, but this doesn’t appear to have a “lazy”
> >>> variant, which I have found to be very beneficial in my use case. The
> >>> Queue is the same one for all processes, if that makes a difference.
> >>>
> >>> I could just make the Queue global, but I have always been told not too.
> >>> Perhaps this is an exception?
> >>
> >> How about wrapping the function into another function that takes only one
> >> argument? A concise way is to do that with functools.partial():
> >>
> >> def f(value, queue): ...
> >>
> >> pool.imap(partial(f, queue=...), values)
> >
> > That looks like exactly what I was looking for. I’ll give it a shot. Thanks!
>
> So as it turns out, this doesn’t work after all. I get an error stating that “Queue objects should only be shared between processes through inheritance”. Still a good technique to know though!
>

Globals aren't as bad as some people think. In this case, a
module-level variable seems like the correct way to do things.

ChrisA