[SciPy-User] Poor scalability of embarrassingly parallel code with multiprocessing

[Rohit Garg]

Hi,

I am attaching a very simple, embarrassingly parallel code. The
problem is that it shows practically no speedup at all on my dual-core
machine.

64 bit, python 2.6,
==============
~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 3
3
done

real	1m5.686s
user	1m41.881s
sys	0m2.846s
~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 2
2
done

real	1m6.052s
user	1m45.550s
sys	0m2.797s
~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 1
1
done

real	1m21.696s
user	1m58.327s
sys	0m2.426s
~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 4
4
done

real	1m3.422s
user	1m39.742s
sys	0m3.179s
========================

And the following tests were done on a quad core machine over ssh, (so
network latencies *may* be part of the problem here).

32 bit, python 2.6,
=============================
~@pixel> time python pool-test.py 1
1
done

real	0m58.772s
user	1m24.549s
sys	0m1.720s
~@pixel> time python pool-test.py 2
2
done

real	0m44.396s
user	1m18.525s
sys	0m2.000s
~@pixel> time python pool-test.py 3
3
done

real	0m45.898s
user	1m21.293s
sys	0m1.904s
~@pixel> time python pool-test.py 4
4
done

real	0m44.196s
user	1m19.561s
sys	0m2.056s
~@pixel> time python pool-test.py 5
5
done

real	0m47.451s
user	1m24.997s
sys	0m2.416s
~@pixel> time python pool-test.py 6
6
done

real	0m49.181s
user	1m25.881s
sys	0m2.356s
==============================

The code which delivered these results is here


===============================
#test of Pool's map capablities
from multiprocessing import Pool
import numpy
import sys

procs=int(sys.argv[1])
print procs
def f(x):
    index,probs=x
    return index,2.0*probs

prob_samples=1000000

probX=numpy.linspace(0.2, 0.3, prob_samples)

Input=[(i,probX[i]) for i in xrange(prob_samples) ]

pool = Pool(processes=procs)

pool.map(f, Input)

print 'done'
===================


Did I make a mistake somewhere? What have been your experiences with
multiprocessing module in general and pool in particular? What
approaches would you suggest to improve the speed scaling.

Regards,
-- 
Rohit Garg

http://rpg-314.blogspot.com/

Senior Undergraduate
Department of Physics
Indian Institute of Technology
Bombay
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pool-test.py
Type: text/x-python
Size: 443 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20091024/090faba9/attachment.py>