[Numpy-discussion] Best way to run python parallel

[Mandus]

On 3/29/07, Brad Malone <brad.malone at gmail.com> wrote:
>
> Hi, I use python for some fairly heavy scientific computations (at least
> to be running on a single processor) and would like to use it in parallel.
> I've seen some stuff online about Parallel Python and mpiPy, but I don't
> know much about them.  Is a python-specific program needed to run python in
> parallel or are the others ( e.g., mpi/poe) just more difficult to work
> with?  And which one would you recommend?



There are, speaking generally, two ways to achive this. One is running a
modified python linked with mpi, the other is using a python module linked
with mpi, which can be used directly in a regular python. In both cases you
will run your application using mpirun, just as with a C/C++ based parallel
program.  Both these approaches have pros and cons, it depends on your
needs.

As you mention, there exist several python mpi wrappers.There are mpi4py at
http://www.cimec.org.ar/python/mpi4py.html (also at http://mpi4py.scipy.org/,
but that page seems to be hacked today). If I started today with Python and
MPI, I probably have checked out mpi4py first as it seems to be most
uptodate - but I haven't tested it myself...  I have used pypar (
http://datamining.anu.edu.au/~ole/pypar/) with success in the past, but I'm
not sure it's updated for numpy. There is also PyMPI (
http://pympi.sourceforge.net), I don't think this is very actively developed
anymore. There are a few more options, I can dig them out if you need more.

You have to watch out for one thing: Some of these implementations can
communicate NumPy arrays directly using low-level C API (basically send/recv
of the data-pointer in a NumPy array), which gives very efficient
communication. Some of them can not do this, and will use pickle to send a
NumPy array - which yields bad performance. Pypar is in the first category
(with NumPy == Numeric, if it's not updated recently...), PyMPI was in the
second category last time I checked. I'm not sure about mpi4py, but I think
it is in the first category - the good one. But watch out for this!

I hope this gives you some pointers at where you can proceed!


-- 
Mandus
The only dr. Mandus around.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070329/0543a1a8/attachment.html>