[py-dev] distributed computing with py.execnet

[holger krekel]

Hi Carl-Friedrich! 

thanks for the feedback! 

On Tue, Jun 21, 2005 at 00:40 +0200, Carl Friedrich Bolz wrote:
> I experimented a bit with py.execnet today and just want to share the 
> result of the experiment. I wrote a little proof of concept for 
> distributing a big computational task between several computers (think 
> cluster) without having to install anything on them.
> 
> There are several processes involved: a server that establishes 
> connections via execnet gateways and sends them their task via the 
> channels. Then the server listens at a socket. If a node has finished 
> its task, it sends the server a notification via the socket and then the 
> server fetches the result via the appropriate channel. Then the node 
> gets a new task via the channel.

In theory and probably even in practice you should not 
have to use a separate socket. What keeps you from (re)using 
a channel there as well?  

For example, you may do: 

    # client (which manages the cluster) 

    def receivernotificationcallback((hostid, result)): 
        # process result coming from hostid (be careful because
        # this callback executes directly in the IO receive
        # thread, you may want to just put things in a queue
        # and process it in some collecting-result thread)

    channel = gw.newchannel(receivernotificationcallback) 

    for hostid in ...: 
        gw.remote_exec(..., channel=channel)  # use preconstructed channel
                                              # and the callback resp.

so the other side's stuff will go the callback function directly. 

If you want to have a channel for control-messages and 
one for results you could do: 

    for hostid in ...: 
        notify_channel = gw.newchannel(receivernotificationcallback) 
        channel = gw.remote_exec(...) 
        channel.send(notify_channel)  
        
the other side can then receive the result channel and sent 
the result when it is ready.  This is probably a cleaner
solution than the former. 

Note that the above usages (and especially the first one) are
a bit on the edge and probably contradict some refactoring ideas i
discussed with Armin.  Mainly, we want to have channels
arbitrarily sendable through a chain of gateways but then
setup/teardown mechanics can become very involved if we allow a
channel to be connected to multiple end-points, i.e. a 1:N relationship. 
Instead we think about restricting channels to always come 
in 1:1 relationships where each end could be send arbitrarily 
across networks and the end points would stay connected to each
other (and routed through the chain of gateways) all the time. 

cheers, 

    holger