[Chicago] multiprocessing map input object creation

[Oren Livne]

Solution: pool.imap()

On 08/13/2013 10:13 AM, Oren Livne wrote:
> Dear All,
>
> I have 20,000 small IbdProblem objects that I'd like to pass to a 
> process pool. Even though I'm passing a generator expression to 
> pool.map, it seems that it hangs for a long time before starting to 
> send them to processes. Maybe all IbdProblem objects are created 
> before processes start working on them. Is that how multiprocessing 
> behaves, or maybe I am missing why it hangs? Is it possible to avoid 
> this and only create each IbdProblem object before it's sent to a 
> process?
>
> Thanks,
> Oren
>
>         manager = Manager()
>         lock = manager.Lock()
>         po = Pool(processes=options.num_processes)
>         res = po.map(process_pair, ((IbdProblem(problem, hap1, hap2, 
> None, params), options, lock)
>                                     for hap1, hap2 in 
> read_pairs(input_file)))
>