Parallelization with Python: which, where, how?

[Michael Hoffman]

Mathias wrote:
> I have a (pretty much) "emberassingly parallel" problem and look for the 
> right toolbox to parallelize it over a cluster of homogenous linux 
> workstations.

We have a >1000-node cluster here and use the commercial Platform LSF to 
manage it. My Poly package 
<http://www.ebi.ac.uk/~hoffman/software/poly/> makes that trivial to use 
from Python and also avoids many of the pitfalls of programming farms 
that large, such as accidental distributed denial of service attacks on 
your own fileserver ;)

Due to the cost and difficulty of setup, LSF is probably not what you 
want, or you would already have it. But MPI is probably not what you 
want if you are doing embarassingly parallelizable problems. I would 
look into OpenPBS <http://www.openpbs.org/>. If you want to write a Poly 
plugin for OpenPBS, I would be happy to accept it. ;)
-- 
Michael Hoffman