[Numpy-discussion] Numeric objects, os.spawnlp and pickle

[Tim Churches]

Aureli Soria Frisch wrote:
> 
> Hi all,
> 
> I am trying to make run a numerical computation (with arrays) in
> different computers simultaneously (in parallel). The computation is
> done under Linux.
> 
> For that purpose a master organizes the process and send rexec
> (remote execute) commands to the different slaves via the python
> command spawnlp. The slaves execute the script specified through
> rexec.
> 
> Inside this script the slaves open a file with the arguments of the
> process, which were serialized via pickle, then make the numerical
> computation, and write the result (a NumPy array) again via pickle in
> a file. This file is opened by the master, which uses the different
> results.
> 
> I am having the problem that the master sometimes (the problem does
> not happen always!!!) open the result and load an object of <type
> 'instance'> instead of the expected object of <type 'array'> (what
> then produces an error). I have tested the type of the objects in the
> slaves and it is always 'array'.
> 
> Has someone made similar experiences by 'pickling' arrays? Could it
> be a problem of the different computers running versions of Python
> from 2.0 to 2.2.1?  Or a problem of different versions of NumPy?
> 
> Is there any other way for doing such a parallel computation?

I am not sure what is causing the unpickling problem you are seeing,
but I suggest that you consider MPI for what you are doing. There are
a number of Python MPI interfaces around, but I can personally recommend
PyPar by Ole Nielsen at the Australian National University. You can use
PyPar with LAM/MPI, which runs in user mode and is very easy to install,
and PyPar itself does not require any modifications to the Python
interpreter.
PyPar will automatically serialise Python objects for you (and
deserialise them
at the destination) but also has methods to send NumPy arrays directly
which is
very efficient. See http://datamining.anu.edu.au/~ole/pypar/ for more
details.

Tim C