[SciPy-User] Speeding things up - how to use more than one computer core

[J. David Lee]

On 04/07/2013 12:25 PM, Gael Varoquaux wrote:
> On Sun, Apr 07, 2013 at 01:11:09PM -0400, Neal Becker wrote:
>>> Regarding the difference between processes and threads:
>> ...
>>> On the other hand, sharing data between threads is much cheaper than
>>> between processes.
>> I have to take issue with this statement.  Sharing data could suffer no
>> overhead at all, if you use shared memory for example.
> How do you use shared memory between processes?
>
> There are solutions, but hardly any are easy to use. I'd even say that
> most are very challenging, and the easiest option is to rely on memapped
> arrays, but even that is a bit technical, and will clearly introduce
> overhead.
I've used shared memory arrays in the past, and it's actually quite 
easy. They can be created using the multiprocessing module in a couple 
of lines,

|mp_arr =multiprocessing.Array(ctypes.c_double,100)
arr = np.frombuffer(mp_arr.get_obj())
|
I've wondered in the past why creating a shared memory array isn't a 
single line of code using numpy, as it can be so useful.

If you can, you might want to consider writing your code in a C module 
and using openMP if it works for you. I've had very good luck with that, 
and it's really easy to use.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20130408/f98e4c35/attachment.html>