multiprocessing & itertools.product Iterator
Kiuhnm
kiuhnm03.4t.yahoo.it
Sun Mar 25 10:52:53 EDT 2012
On 3/25/2012 0:35, Christian wrote:
> Hey,
>
> I struggle to "extend" a multiprocessing example to my problem with a
> itertools.product result iterator.
> How I have to assign the combos.next() elements approriate to
> Pool.imap/calc functions?
>
> Thanks in advance
> Christian
>
>
> from multiprocessing import Process,Queue,Pool
> import Calculation
> import DataSimulation
> from itertools import product
>
>
> def produce_example_combos(size=6,na=1000,nb=10):
> data = DataSimulation.DataSimulation()
> a = [data.generate_simple(size) for x in xrange(na)]
> b = [data.generate_simple(size) for x in xrange(nb)]
> it = product(a,b)
> return it
>
> def calc(elements):
> calc.q.put("Doing:" + elements[0] + elements[1])
This throws because elements[0] isn't a string.
Try with
calc.q.put("Doing: {} {}".format(elements[0], elements[1]))
or something similar.
To see the error, use something like
try:
calc.q.put("Doing:" + elements[0] + elements[1])
ratio = Calculation.ratio(elements[0],elements[1])
return ratio
except Exception as exc:
print(exc.__doc__)
> ratio = Calculation.ratio(elements[0],elements[1])
> return ratio
>
> def calc_init(q):
> calc.q = q
>
>
> if __name__ == '__main__':
> combos = produce_example_combos()
> print "tesdata generated"
> q = Queue()
> p = Pool(10, calc_init, [q])
> results = p.imap(calc,combos.next())
Why combos.next()?
imap expects an iterable, i.e. combos, not the first element in combos.
That's similar to
for i in combos.next()
> p.close()
I don't know whether p.wait() is also needed here.
> for i in combos:
> print q.get()
> print results.next()
That doesn't work, because combos is an iterator and you've already got
to the end (next() after next()).
Look at this example:
combos = produce_example_combos()
for i in combos:
print("ok")
for i in combos:
print("not ok")
That code will print a few "ok" but not even a single "not ok".
You should write
for r in results:
print q.get()
print r
Here's a working example (I had to do some editing):
---->
from multiprocessing import Process,Queue,Pool
#import Calculation
#import DataSimulation
from itertools import product
def produce_example_combos(size=6,na=1000,nb=10):
#data = DataSimulation.DataSimulation()
#a = [data.generate_simple(size) for x in xrange(na)]
#b = [data.generate_simple(size) for x in xrange(nb)]
a = ['a', 'b', 'c']
b = [1, 2, 3]
it = product(a,b)
return it
def calc(elements):
calc.q.put("Doing: {}{}".format(elements[0], elements[1]))
#ratio = Calculation.ratio(elements[0], elements[1])
ratio = elements
return ratio
def calc_init(q):
calc.q = q
if __name__ == '__main__':
combos = produce_example_combos()
print("tesdata generated")
q = Queue()
p = Pool(10, calc_init, [q])
results = p.imap(calc, combos)
p.close()
p.join()
for r in results:
print(q.get())
print(r)
input("")
<----
Kiuhnm
More information about the Python-list
mailing list