[Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

[Sturla Molden]

Benjamin Root <ben.v.root at gmail.com> wrote:

> Oftentimes, if one needs to share numpy arrays for multiprocessing, I would
> imagine that it is because the array is huge, right? 

That is a case for shared memory, but what. i was taking about is more
common than this. In order for processes to cooperate, they must
communicate. So we need a way to pass around NumPy arrays quickly.
Sometimes we want to use shared memory because of the size of the data, but
more often it is just used as a form of inexpensive IPC.

> So, the pickling
> approach would copy that array for each process, which defeats the purpose,
> right?

I am not sure what you mean. When I made shared memory arrays I used named
segments, and made sure only the name of the segments were pickled, not the
contents of the buffers.

Sturla