[SciPy-User] Shared memory using multiprocessing.sharedctypes

[David Baddeley]

Here's the code, hope it's useful. 

cheers,
David

--- On Tue, 16/3/10, Charles Vejnar <ce at vejnar.eu> wrote:

> From: Charles Vejnar <ce at vejnar.eu>
> Subject: Re: [SciPy-User] Shared memory using multiprocessing.sharedctypes
> To: "SciPy Users List" <scipy-user at scipy.org>
> Cc: "David Baddeley" <david_baddeley at yahoo.com.au>
> Received: Tuesday, 16 March, 2010, 4:31 AM
> Hi David,
> 
> I am interested. Can you please show us your code ?
> 
> BW,
> Charles
> 
> On Monday 15 March 2010, David Baddeley wrote:
> > Thanks!
> > 
> > Will have to give it a spin. Unfortunately  (or
> fortunately - as it was
> >  quite instructive from a learning perspective) I
> got bored and already
> >  knocked up my own version. It delegates all the
> memory management to
> >  multiprocessing so weighs in pretty light at
> only 80 lines of pure python
> >  & works well on 64 bit systems. Just in case
> anyone wants yet another
> >  shared memory array implementation.
> > 
> > cheers,
> > David
> > 
> > 
> > ----- Original Message ----
> > From: Christopher Lee-Messer <chris.leemesser at gmail.com>
> > To: scipy-user at scipy.org
> > Sent: Sun, 14 March, 2010 5:45:08 AM
> > Subject: [SciPy-User]  Shared memory using
> multiprocessing.sharedctypes
> > 
> > David Baddeley wrote::
> > > I'd like shared memory numpy arrays for some code
> I'm trying to
> > > parallelise. The solution should ideally be cross
> platform. I remember
> > > some earlier discussion about this and, as far as
> I recall Sturla had a
> > > working implementation which was (unfortunately)
> linux only (please
> > > correct me if I'm wrong).
> > >
> > > Looking a bit further,
> multiprocessing.sharedctypes seems to have all the
> > > stuff needed to handle the low level bits on at
> least linux and windows,
> > > and it seems as though it should be relatively
> easy to cobble something
> > > together on top of this which let you use shared
> memory numpy arrays
> > > relatively transparently. This would have the
> advantage that all the
> > > platform specific nasty stuff is being handled by
> the multiprocessing
> > > module, which as it's now part of the standard
> library, should hopefully
> > > be well maintained & has a good chance of
> being ported to additional
> > > platforms
> > >
> > > Before I jump in and try writing such a wrapper -
> essentially a helper
> > > function for creating shared numpy arrays (easy)
> and some pickle handlers
> > > to make sure what comes out the other end looks
> like an array (would need
> > > to think/read a bit more, but should be doable) I
> wanted to make sure
> > > that there's not something out there already
> & that I'm not going to be
> > > reinventing the wheel.
> > 
> > [Note, I posted this via google groups originally
> before realizing
> > that it wasn't the official list. Sorry if you get
> multiple versions.
> > -chris]
> > 
> > Hi David,
> > I was just in the same boat.  I checked out
> Sturla's a few weeks ago
> > and tried it out on windows  xp and now linux
> (32bit, python 2.6 with
> > numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with
> numpy 1.2.1)
> > 
> > It's working very well for me in soft-realtime data
> acquisition
> > application with sizes of 1 to 800KB.
> > 
> > In case it saves you time, I recently threw together
> some packaging
> > for the code and put it on bitbucket as a
> numpy-sharedmem repository
> > in case it might save me or someone else some time.
> > 
> > http://bitbucket.org/cleemesser/numpy-sharedmem/
> > 
> > I don't know if Sturla, Gael Varoquaux, or Robert Kern
> are continuing
> > to work on these, but I plan to add testing and save
> results on
> > different platforms as my app gets used in different
> computers in the
> > lab.  Nadav Horesh mentioned that there are some
> limitations on the
> > size of the arrays for "large" arrays and Sturla
> mentioned that there
> > may be issue on 64bit systems.
> > -Chris
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> > 
> > 
> > 
> > 
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> > 
>


      
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shmarray.py
Type: text/x-python
Size: 5246 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/eab3fd02/attachment.py>