[SciPy-User] Multiprocessing and shared memory

[Sturla Molden]

Felix Schlesinger skrev:
> I have been working on an application using scipy that solves a highly
> parallel problem. To avoid the GIL in python I used to multiprocessing
> package. The main issue I ran into is shared memory.

Gaël Varoquaux and I did some work on that some months ago. It's not as 
trivial as it seems, but we have a working solution.

http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip

Basically it uses named shared memory (Sys V IPC on Unix) as buffer. The 
ndarray is pickled by its kernel name, the buffer is not copied. Thus 
you can quickly communicate shared memory ndarrays between processes 
(using multiprocessing.Queue).

Note that there is a pesky memory we could get rid of on Unix: It stems 
from multiprocessing using os._exit instead of sys.exit to terminate 
worker processes, preventing any clean-up code from executing. The bug 
is in multiprocessing, not our code. Windows refcounts shared memory 
segments in the kernel and is not affected. Change any occurrence of 
os._exit in multiprocessing to sys.exit and it will work just fine.

Well, it's not quite up to date for all 64 bits systems. I'll fix that 
some day.


Sturla Molden