[SciPy-user] shared memory machines

[Sturla Molden]

On 2/9/2009 4:07 PM, Philip Semanchuk wrote:

> Unfortunately POSIX IPC doesn't report that information.

I'll suggest we use System V IPC instead, as it does report a ref count. 
Code example attached. It compiles with Cython but I have not done any 
testing except that.

My suggestion is to spawn a thread in the creator process to monitor the 
attachment count for the segment, and mark it for removal when it has 
dropped to zero. There is a __dealloc__ in a Handle object that does the 
shmdt, and then Python should do the refcounting (similar to what is 
done for CloseHandle in Windows).

We have to figure out what to do with ctrl-c. It is a source of trouble. 
With a daemonic GC thread it could cause a leak, with a non-daemonic GC 
thread it may hang forever (which is also a leak). So I opted for a 
daemonic GC thread.

I also have a version of the Windows sharedmem with a small bugfix (I 
forgot to unmap the segment before closing the handle). I had to remove 
the mutex from the Windows code. It can be put in a separate module. We 
should also have a lock with a named Sys V semaphore.

> Since I'm not a numpy user I'm a little lost as to how you're using  
> the shared memory here, but I gather that it is effectively "magic" to  
> a numpy user? i.e., he doesn't have any idea that a shared memory  
> segment is being created on his behalf? If that's the case I don't see  
> any way around reference counting.

We are going to use multiple processes as if they were threads. It is 
basically a hack to work around Python's GIL (global interpreter lock). 
Basically we want to create ndarray's with the same interface as before, 
except that they have shared memory as data. For example,

import numpy
a = numpy.zeros((4,1024), order='F', dtype=float)

import scipy
a = scipy.sharedmem.zeros((4,1024), order='F', dtype=float)

should do the same, except that the latter uses shared memory. And when 
it is sent through a multiprocessing.Queue, only the segment name, 
offset, shape and dtype gets pickled. In the former case, a copy of the 
whole data buffer is made. Right now we are just creating the shared 
memory buffer to use as backend.

In multiprocessing you will find an object called mp.Array. We can wrap 
its buffer with an ndarray, but it cannot be passes through a mp.Queue. 
In other words, all shared memory must be allocated in advance. And that 
is what we don't want.


Sturla Molden
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sharedmemory_sysv.pyx
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20090209/92a36f26/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sharedmemory_win.pyx
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20090209/92a36f26/attachment-0001.ksh>