[SciPy-User] Shared memory using multiprocessing.sharedctypes

[Charles Vejnar]

Hi David,

I am interested. Can you please show us your code ?

BW,
Charles

On Monday 15 March 2010, David Baddeley wrote:
> Thanks!
> 
> Will have to give it a spin. Unfortunately  (or fortunately - as it was
>  quite instructive from a learning perspective) I got bored and already
>  knocked up my own version. It delegates all the memory management to
>  multiprocessing so weighs in pretty light at only 80 lines of pure python
>  & works well on 64 bit systems. Just in case anyone wants yet another
>  shared memory array implementation.
> 
> cheers,
> David
> 
> 
> ----- Original Message ----
> From: Christopher Lee-Messer <chris.leemesser at gmail.com>
> To: scipy-user at scipy.org
> Sent: Sun, 14 March, 2010 5:45:08 AM
> Subject: [SciPy-User]  Shared memory using multiprocessing.sharedctypes
> 
> David Baddeley wrote::
> > I'd like shared memory numpy arrays for some code I'm trying to
> > parallelise. The solution should ideally be cross platform. I remember
> > some earlier discussion about this and, as far as I recall Sturla had a
> > working implementation which was (unfortunately) linux only (please
> > correct me if I'm wrong).
> >
> > Looking a bit further, multiprocessing.sharedctypes seems to have all the
> > stuff needed to handle the low level bits on at least linux and windows,
> > and it seems as though it should be relatively easy to cobble something
> > together on top of this which let you use shared memory numpy arrays
> > relatively transparently. This would have the advantage that all the
> > platform specific nasty stuff is being handled by the multiprocessing
> > module, which as it's now part of the standard library, should hopefully
> > be well maintained & has a good chance of being ported to additional
> > platforms
> >
> > Before I jump in and try writing such a wrapper - essentially a helper
> > function for creating shared numpy arrays (easy) and some pickle handlers
> > to make sure what comes out the other end looks like an array (would need
> > to think/read a bit more, but should be doable) I wanted to make sure
> > that there's not something out there already & that I'm not going to be
> > reinventing the wheel.
> 
> [Note, I posted this via google groups originally before realizing
> that it wasn't the official list. Sorry if you get multiple versions.
> -chris]
> 
> Hi David,
> I was just in the same boat.  I checked out Sturla's a few weeks ago
> and tried it out on windows  xp and now linux (32bit, python 2.6 with
> numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with numpy 1.2.1)
> 
> It's working very well for me in soft-realtime data acquisition
> application with sizes of 1 to 800KB.
> 
> In case it saves you time, I recently threw together some packaging
> for the code and put it on bitbucket as a numpy-sharedmem repository
> in case it might save me or someone else some time.
> 
> http://bitbucket.org/cleemesser/numpy-sharedmem/
> 
> I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
> to work on these, but I plan to add testing and save results on
> different platforms as my app gets used in different computers in the
> lab.  Nadav Horesh mentioned that there are some limitations on the
> size of the arrays for "large" arrays and Sturla mentioned that there
> may be issue on 64bit systems.
> -Chris
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>