[SciPy-user] shared memory machines

[Sturla Molden]

> On Sun, Feb 01, 2009 at 10:03:30AM -0500, Gideon Simpson wrote:

> Actually I wrote this code a few months ago, and now that I am looking at
> it, I realise that the SharedMemArray should probably be a subclass of
> numpy.ndarray, and implement the full array signature. I am not sure if
> this is possible or not (ie if it will still be easy to have
> multiprocessing share the data between processes or not). ¨

You can use multiprocessing.Array to allocate shared memory, and use its
buffer to create an ndarray with numpy.frombuffer.

Basically multiprocessing can use whatever can be pickled. ndarrays copy
their contents when pickled, and subclasses seem to inherit this
behaviour. Note that this is perfectly okay if you are happy with a
message-passing approach to parallel computing.

When using mp.Array as shared memory, the object must be passed to
multiprocessing.Process on instantiation. This is because of handle
inheritance. Therefore you cannot pass an instance of mp.Array through a
mp.Queue or mp.Pipe. If we use named share memory (System V IPC) instead
of BSD mmap, we can probably get around this. But support for this li
lacking in Python and SciPy.

Sturla Molden