[Numpy-discussion] Best way to run python parallel

[Brian Granger]

A few replies have already given you some idea of the options.  A
comments to supplement those made so far:

1.  mpi4py.  I have used this quite a bit and it is incredible.  It
seems to build just about anywhere, and it has very complete coverage
of the mpi spec - even the mpi-2 stuff.  In fact, the coverage of the
MPI spec is so good that the test suite for mpi4py has uncovered
_many_ bugs in the various underlying MPI implementations.  mpi4py
also handles numpy arrays well.  Lisandro Dalcin (author of mpi4py)
has written a paper that has performance benchmarks.  Amazingly,
mpi4py has almost the same performance as low-level C code when numpy
arrays are passed around.  If you need to use MPI, you will be very
happy that you used mpi4py.

2.  IPython1.  As Fernando mentioned, we have been doing substantial
work to enable interactive parallel computing with IPython.  The
currently stable version of this work is documented here:

http://ipython.scipy.org/moin/Parallel_Computing

People are using this today in production codes.  We are, however,
busy working on a new release, called "saw" - it should be out in the
next 2 weeks and has many new features and improvements over the
current release.  There is a talk that I gave at PyCon about IPython1
here:

http://ipython.scipy.org/moin/About/Presentations

The most important question you for though, is which of these tools
should you use?  Here is my take.

If you have a problem that can be trivially parallelized (meaning the
different processors don't have to communicate much), then IPython1 is
going to be your best bet.  It will allow you to do this type of stuff
quickly and easily - and in a fully interactive way.  The new version
of IPython1 (saw) that is coming out soon even has a nice load
balanced task farming system that makes this type of stuff _super_
easy.

If, however, you need to send lots of data between the processors, you
should absolutely use MPI..  I highly recommend using OpenMPI for your
MPi implementation and using mpi4py on top of that.  The best part is
that IPython1 has been designed to play well with mpi4py (and the
other python MPI bindings).  Thus, you can write regular MPI code and
then run/debug/control it interactively using IPython1.

Please let us know if you have other questions or need help getting
going with IPython1 or mpi4py.

Cheers,

Brian




On 3/29/07, Brad Malone <brad.malone at gmail.com> wrote:
> Hi, I use python for some fairly heavy scientific computations (at least to
> be running on a single processor) and would like to use it in parallel.
> I've seen some stuff online about Parallel Python and mpiPy, but I don't
> know much about them.  Is a python-specific program needed to run python in
> parallel or are the others ( e.g., mpi/poe) just more difficult to work
> with?  And which one would you recommend?
>
> Thanks for your time, I appreciate it.
>
> Brad
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>