[Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

[Sturla Molden]

Den 16.06.2011 18:44, skrev Christopher Barker:
>
> I'll bet this is a big issue, and one I'm curious about how to 
> address, I have another problem where I need to multi-process, and I'd 
> love to know a way to pass data to the other process and back 
> *without* going through pickle. maybe memmapped files?

Remember that os.fork is copy-on-write optimized. Often this is enough 
to share data.

To get data back, a possibility is to use shared memory: just memmap 
from file 0 or -1 (depending on the system) and fork.

Note that the use of fork, as opposed to multiprocessing, avoids the 
pickle overhead for NumPy arrays.

On Windows this sucks, because there is no fork system call. Here we are 
stuck with multiprocessing and pickle, even if we use shared memory.

Sturla