[Numpy-discussion] Shared memory ndarrays (update)
Sturla Molden
sturla at molden.no
Sun Apr 10 19:20:39 EDT 2011
Here is an update for the shared memory arrays that Gaël and I wrote two
years ago. They are NumPy arrays referencing shared memory, and IPC
using multiprocessing.Queue is possible by monkey patching how ndarrays
are pickled.
Usage:
import numpy as np
import sharedmem as sm
shared_array = sm.zeros(n)
I.e. the only difference from ndarrays is that pickle.dumps and
multiprocessing.Queue do not make a copy of the buffer, and that
allocated memory is shared between professes (e.g. created with os.fork,
subprocess or multiprocessing.)
A named memory map of the paging file is used on Windows. Unix System V
IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance).
Changes:
- 64-bit support.
- Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
- Added a global lock as there are callbacks to Python (the GIL is not
sufficient serialization).
I need help with testing, particularly on Linux / Apple and with the
most recent NumPy.
I'm an idiot with build tools, hence no setup.py. Invoke Cython and then
cc. Compile sharedmemory_sysv.pyx for Linux/Unix or
sharedmemory_sysv.pyx and ntqueryobject.c for Windows.
Regards,
Sturla
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sharedmem.zip
Type: application/x-zip-compressed
Size: 8794 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/a6b16acb/attachment.bin>
More information about the NumPy-Discussion
mailing list