[SciPy-User] Shared memory using multiprocessing.sharedctypes

[Christopher Lee-Messer]

David Baddeley wrote::
> I'd like shared memory numpy arrays for some code I'm trying to parallelise. The solution should ideally be cross platform. I remember some earlier discussion about this and, as far as I recall Sturla had a working implementation which was (unfortunately) linux only (please correct me if I'm wrong).
>
> Looking a bit further, multiprocessing.sharedctypes seems to have all the stuff needed to handle the low level bits on at least linux and windows, and it seems as though it should be relatively easy to cobble something together on top of this which let you use shared memory numpy arrays relatively transparently. This would have the advantage that all the platform specific nasty stuff is being handled by the multiprocessing module, which as it's now part of the standard library, should hopefully be well maintained & has a good chance of being ported to additional platforms
>
> Before I jump in and try writing such a wrapper - essentially a helper function for creating shared numpy arrays (easy) and some pickle handlers to make sure what comes out the other end looks like an array (would need to think/read a bit more, but should be doable) I wanted to make sure that there's not something out there already & that I'm not going to be reinventing the wheel.
>
[Note, I posted this via google groups originally before realizing
that it wasn't the official list. Sorry if you get multiple versions.
-chris]

Hi David,
I was just in the same boat.  I checked out Sturla's a few weeks ago
and tried it out on windows  xp and now linux (32bit, python 2.6 with
numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with numpy 1.2.1)

It's working very well for me in soft-realtime data acquisition
application with sizes of 1 to 800KB.

In case it saves you time, I recently threw together some packaging
for the code and put it on bitbucket as a numpy-sharedmem repository
in case it might save me or someone else some time.

http://bitbucket.org/cleemesser/numpy-sharedmem/

I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
to work on these, but I plan to add testing and save results on
different platforms as my app gets used in different computers in the
lab.  Nadav Horesh mentioned that there are some limitations on the
size of the arrays for "large" arrays and Sturla mentioned that there
may be issue on 64bit systems.
-Chris