Parallel Python

[Konrad Hinsen]

On Jan 8, 2007, at 11:33, Duncan Booth wrote:

> The 'parallel python' site seems very sparse on the details of how  
> it is
> implemented but it looks like all it is doing is spawning some  
> subprocesses
> and using some simple ipc to pass details of the calls and results.  
> I can't
> tell from reading it what it is supposed to add over any of the other
> systems which do the same.
>
> Combined with the closed source 'no redistribution' license I can't  
> really
> see anyone using it.

I'd also like to see more details - even though I'd probably never  
use any Python module distributed in .pyc form only.

 From the bit of information there is on the Web site, the  
distribution strategy looks quite similar to my own master-slave  
distribution model (based on Pyro) which is part of ScientificPython.  
There is an example at

http://dirac.cnrs-orleans.fr/hg/ScientificPython/main/? 
f=08361040f00a;file=Examples/master_slave_demo.py

and the code itself can be consulted at

http://dirac.cnrs-orleans.fr/hg/ScientificPython/main/? 
f=bce321680116;file=Scientific/DistributedComputing/MasterSlave.py


The main difference seems to be that my implementation doesn't start  
compute jobs itself; it leaves it to the user to start any number he  
wants by any means that works for his setup, but it allows a lot of  
flexibility. In particular, it can work with a variable number of  
slave jobs and even handles disappearing slave jobs gracefully.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: hinsen at cnrs-orleans.fr
---------------------------------------------------------------------