[Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

[Sturla Molden]

Feng Yu <rainwoodman at gmail.com> wrote:

> 1. If we are talking about shared memory and copy-on-write
> inheritance, then we are using 'fork'. 

Not available on Windows. On Unix it only allows one-way communication,
from parent to child.

> 2. Picking of inherited shared memory array can be done minimally by
> just picking the array_interface and the pointer address. It is
> because the child process and the parent share the same address space
> layout, guarenteed by the fork call.

Again, not everyone uses Unix. 

And on Unix it is not trival to pass data back from the child process. I
solved that problem with Sys V IPC (pickling the name of the segment).

> 6. If we are to define a set of operations I would recommend take a
> look at OpenMP as a reference -- It has been out there for decades and
> used widely. An equiavlant to the 'omp parallel for' construct in
> Python will be a very good starting point and immediately useful.

If you are on Unix, you can just use a context manager. Call os.fork in
__enter__ and os.waitpid in __exit__. 

Sturla