[Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

[Feng Yu]

>
> Personally I prefer a parallel programming style with queues – either to
> scatter arrays to workers and collecting arrays from workers, or to chain
> workers together in a pipeline (without using coroutines). But exactly how
> you program is a matter of taste. I want to make it as inexpensive as
> possible to pass a NumPy array through a queue. If anyone else wants to
> help improve parallel programming with NumPy using a different paradigm,
> that is fine too. I just wanted to clarify why I stopped working on shared
> memory arrays.

Even I am not very obsessed with functional and queues, I still have
to agree with you
queues tend to produce more readable and less verbose code -- if there
is the right tool.

>
> (As for the implementation, I am also experimenting with platform dependent
> asynchronous I/O (IOCP, GCD or kqueue, epoll) to pass NumPy arrays though a
> queue as inexpensively and scalably as possible. And no, there is no public
> repo, as I like to experiment with my pet project undisturbed before I let
> it out in the wild.)

It will be wonderful if there is a way to pass numpy array around
without a huge dependency list.

After all, we know the address of the array and, in principle we are
able to find the physical pages and map them in the receiver side.

Also, did you checkout http://zeromq.org/blog:zero-copy ?
ZeroMQ is a dependency of Jupyter, so it is quite available.

- Yu

>

>
> Sturla
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion