Running a python farm

[Andrew Bennetts]

On Mon, Oct 27, 2003 at 02:28:05PM +0000, Ian McConnell wrote:
> What's the pythonic way of sending out a set of requests in parallel?
> 
> My program throws an image at the server and then waits for the result. I'm
> currently using this bit of socket code to send an image to server on
> another machine.
> 
> 
> import socket
> import SocketServer
> import cPickle
> 
> def oneclient(host, port, array):
>     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>     sock.connect((host, port))
>     sent = sock.sendall(cPickle.dumps(array, 0))
>     sock.shutdown(1)
>     rfile = sock.makefile('rb')
>     return cPickle.loads(rfile.read())
> 
> 
> Processing the image can take several minutes and I have loads of images to
> process, but I also have several servers available, so I'd like save some
> time by distributing the images around the servers. So for 'n' servers,
> throw 'n' images at them. Then as each server finishes, give it another
> image to work on until I've done all the images.

Probably the simplest way is to have n worker threads, and a main thread
that pushes the images to be processed onto a Queue.Queue instance.

So, your worker thread function might look something like (using your
function above):

    def worker(host, port, queue):
        while 1:
            array = queue.get()
            if array is None:
                return
            result = oneclient(host, port, array)
            # do something with the result, perhaps store in a shared
            # list or dictionary?

Then you just need a main function like:

    import Queue, threading
    
    servers = [('host1', 1111), ('host2', 2222)]  # etc...
    
    def main():
        q = Queue.Queue()
        workers = []
        for host, port in servers:
            t = threading.Thread(target=worker, args=(host, port, q))
            t.start()
            workers.append(t)

        # assume we get arrays to be processed from somewhere
        for array in arrays:
            q.put(array)

        # Tell the worker threads to stop when there's nothing left to do
        for i in range(len(workers)):
            q.put(None)

        # Wait for all the treads to finish
        for t in workers:
            t.join()

        # Do something with the results gathered by the workers
        pass

-Andrew.