Inter Process Communication with Numeric arrays

[Tim Churches]

Ian McConnell wrote:
> 
> I want to send the results of some calculation on a Numeric array to a
> second process. These arrays will be fairly large (about 1Mb) so can anyone
> recommend a package for simple IPC?
> 
> I'm writing both processes (though one might be in C) so ORB/CORBA seems to
> be overkill and I can't get through to the memory mapped files at
>         http://pobox.com/~kragen/sw/arrayfrombuffer/
> (Internal server error)
> 
> Suggestions?

Have you thought about using MPI? MPI is usually used for cluster
computing but
it also works well for interprocess communication on a single machine -
it is 
commonly used for such purposes on multi-processor SMP machines, but you
can tell
MPI how many processes you want even on a single CPU machine. If you use
LAM/MPI,
which runs in user mode, it is a snap to install and you don't have to
recompile your kernel
- just run it when you need it. Because MPI was written with high
perfromance computing in mind,
it is very efficient. And if you ever want to split your processes
across different machines,
MPI will handle sending the data across the network transparently. My
understanding is that
MPI uses in-memory IPC where it can (on a shared-memory machine),
otherwise it uses network
transport layers - but it looks after all the messy details.

But best of all, Ole Nielsen at the Australian National University has
written a very
nice Python wrapper for MPI called PyPar which makes sending Python data
structures between
processes a snap, and it will even send NumPy arrays directly, without
serialisation. And you
don't need to modify your Python interpreter to use it - just import the
module.
See http://datamining.anu.edu.au/~ole/pypar/ for details.

Tim C