Pool module -- does map pre-assign tasks to processes?

Aahz aahz at pythoncraft.com
Mon Oct 5 14:14:35 EDT 2009


In article <a6593d46-0f0e-4ca7-8481-c53b18218f57 at o13g2000vbl.googlegroups.com>,
Luca  <luca.de.alfaro at gmail.com> wrote:
>I would like to use the Pool module, but my tasks take sometimes
>unpredictably different time to complete.  The simplest way to write
>the code would be to put all task descriptions in an array, then call
>
>p = Pool(8)
>p.map(f, a)
>
>But I don't want to preassign process 0 with elements a[0], a[8], a
>[16],  process 1 with elements a[1], a[9], a[17], and so forth.
>Rather, I would like all tasks to be put into a queue, and I would
>like the processes to each time grab the next task to be done, and do
>it.  This would ensure fairly equal loading.
>
>My question is: does the map() method of Pool pre-assign which task
>gets done by each process, or is this done at runtime, on a get-first-
>task-to-be-done basis?

You can probably get what you want by setting chunksize to 1.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"....Normal is what cuts off your sixth finger and your tail..."  --Siobhan



More information about the Python-list mailing list