Running a python farm

[Ian Sparks]

Ian McConnell wrote:
>>
What's the pythonic way of sending out a set of requests in parallel?

My program throws an image at the server and then waits for the result. I'm
currently using this bit of socket code to send an image to server on
another machine.

...

What should I be looking for, for a pythonic solution? I'm stuck for
terminology? Threads? Load balancing? Processors farms? Any pointers or
suggestions welcome.
<<


This sounds like a perfect job for a TupleSpace.

A TupleSpace is basically a bucket with a set of simple operations : Put, Take (and sometimes Read).

A set of processes (normally connecting from different machines) make requests of the tuplespace which block until a tuple arrives that matches their request. When a matching tuple arrives it takes it out of the bucket.

This makes it very easy to add/remove PC's to a "renderfarm" where each processor takes work units out of the bucket and when finished puts the result into the bucket.

Java has a nice TupleSpace implementation going by the name of JavaSpaces. I believe that PyBrenda is the closest thing Python has ever had to a TupleSpace but I think its a dead project now.

I have never used it but this thread purports to have created a simple TupleSpace using the Quixote web framework :

http://mail.mems-exchange.org/pipermail/quixote-users/2003-July/001776.html

I think a general-purpose, probably sockets-based TupleSpace implementation for Python would be a very cool thing.....

Don't know if this will help your problem but I hope your request will raise the profile of TupleSpaces in the Python community.

- Ian Sparks.





-----Original Message-----
From: Ian McConnell [mailto:ian at emit.demon.co.ukx]
Sent: Monday, October 27, 2003 9:28 AM
To: python-list at python.org
Subject: Running a python farm


What's the pythonic way of sending out a set of requests in parallel?

My program throws an image at the server and then waits for the result. I'm
currently using this bit of socket code to send an image to server on
another machine.


import socket
import SocketServer
import cPickle

def oneclient(host, port, array):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((host, port))
    sent = sock.sendall(cPickle.dumps(array, 0))
    sock.shutdown(1)
    rfile = sock.makefile('rb')
    return cPickle.loads(rfile.read())


Processing the image can take several minutes and I have loads of images to
process, but I also have several servers available, so I'd like save some
time by distributing the images around the servers. So for 'n' servers,
throw 'n' images at them. Then as each server finishes, give it another
image to work on until I've done all the images.

What should I be looking for, for a pythonic solution? I'm stuck for
terminology? Threads? Load balancing? Processors farms? Any pointers or
suggestions welcome.



I don't have to do this too often, so I would prefer a simple solution over
performance.


Thanks.


-- 
 "Thinks: I can't think of a thinks. End of thinks routine": Blue Bottle

** Aunty Spam says: Remove the trailing x from the To: field to reply **
-- 
http://mail.python.org/mailman/listinfo/python-list