[Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

[Brandt Belson]

Hello all,
Since I haven't heard back, I decided to simplify my problem even more. I'm
attaching two files which demonstrate the problem very clearly. My original
message is below. The script can be run with "python shared_mem.py".
Thanks,
Brandt


> > Hello,
> > I'm parallelizing some code I've written using the built in
> multiprocessing
> > module. In my application, I need to multiply many large arrays together
> > and
> > sum the resulting product arrays (inner products). I noticed that when I
> > parallelized this with myPool.map(...) with 8 processes (on an 8-core
> > machine), the code was actually about an order of magnitude slower. I
> > realized that this only happens when I use the numpy array
> multiplication.
> > I
> > defined my own array multiplication and summation. As expected, this is
> > much
> > slower than the numpy versions, but when I parallelized it I saw the
> > roughly
> > 8x speedup I expected. So something about numpy array multiplication
> > prevented me from speeding it up with multiprocessing.
> >
> > Is there a way to speed up numpy array multiplication with
> multiprocessing?
> > I noticed that numpy's SVD uses all of the available cores on its own,
> does
> > numpy array multiplication do something similar already?
> >
> > I'm copying the code which can reproduce and summarize these results.
> > Simply
> > run: "python shared_mem.py" with myutil.py in the same directory.
> >
> > Thanks,
> > Brandt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110609/93d27d7c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shared_mem.py
Type: application/octet-stream
Size: 1522 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110609/93d27d7c/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: myutil.py
Type: application/octet-stream
Size: 384 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110609/93d27d7c/attachment-0001.obj>