Distributed computation of jobs (was: Parallel Python)

[A.T.Hofkamp]

On 2007-01-12, robert <no-spam at no-spam-no-spam.invalid> wrote:
>> 
>> [1] http://www.python.org/pypi/parallel
>
> I'd be interested in an overview.
> For ease of use a major criterion for me would be a pure python 
> solution, which also does the job of starting and controlling the 
> other process(es) automatically right (by default) on common 
> platforms.

Let me add a few cents to the discussion with this announcement:

About three years ago, I wrote two Python modules, one called 'exec_proxy',
which uses ssh to run another exec_proxy instance at a remote machine, thus
providing ligh-weight transparent access to a machine across a network.

The idea behind this module was/is that by just using ssh you have network
transparency, much more light weight than most other distributed modules where
you have to start deamons at all machines.
Recently, the 'rthread' module was announced which takes the same approach (it
seems from the announcement). I have not compared both modules with each other.


The more interesting Python module called 'batchlib' lies on top of the former
(or any other module that provides transparency across the network). It
handles distribution of computation jobs in the form of a 'start-computation'
and 'get-results' pair of functions.

That is, you give it a set of machines it may use, you say to the entry-point,
compute for me this-and-this function with this-and-this parameters, and
batchlib does the rest.
(that is, it finds a free machine, copies the parameters over the network, runs
the job, the result is transported back, and you can get the result of a
computation by using the same (uniq) identification given by you when the job
was given to batchlib.)

We used it as computation backend for optimization problems, but since
'computation job' may mean anything, the module should be very generically
applicable.


Compared to most other parallel/distributed modules, I think that the other
modules more-or-less compare with exec_proxy (that is, they stop with
transparent network access), where exec_proxy was designed to have minimal
impact on required infra structure (ie just ssh or rsh which is generally
already available) and thus without many of the features available from the
other modules.

Batchlib starts where exec_proxy ends, namely lifting network primitives to the
level of providing a simple way of doing distributed computations (in the case
of exec_proxy, without adding network infra structure such as deamons).




Until now, both modules were used in-house, and it was not clear what we wanted
to do further with the software. Recently, we have decided that we have no
further use for this software (we think we want to move into a different
direction), clearing the way to release this software to the community.

You can get the software from my home page http://seweb.se.wtb.tue.nl/~hat
Both packages can be downloaded, and include documentation and an example.
The bad news is that I will not be able to do further development of these
modules. The code is 'end-of-life' for us.


Maybe you find the software useful,
Albert