Running a python farm

[John Roth]

"Ian McConnell" <ian at emit.demon.co.ukx> wrote in message
news:877k2qrc0a.fsf at emit.demon.co.uk...
> What's the pythonic way of sending out a set of requests in parallel?
>
> My program throws an image at the server and then waits for the result.
I'm
> currently using this bit of socket code to send an image to server on
> another machine.
>
>
> import socket
> import SocketServer
> import cPickle
>
> def oneclient(host, port, array):
>     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>     sock.connect((host, port))
>     sent = sock.sendall(cPickle.dumps(array, 0))
>     sock.shutdown(1)
>     rfile = sock.makefile('rb')
>     return cPickle.loads(rfile.read())
>
>
> Processing the image can take several minutes and I have loads of images
to
> process, but I also have several servers available, so I'd like save some
> time by distributing the images around the servers. So for 'n' servers,
> throw 'n' images at them. Then as each server finishes, give it another
> image to work on until I've done all the images.
>
> What should I be looking for, for a pythonic solution? I'm stuck for
> terminology? Threads? Load balancing? Processors farms? Any pointers or
> suggestions welcome.
>
>
>
> I don't have to do this too often, so I would prefer a simple solution
over
> performance.

All the solutions I know of have adequate performance. Andrew's
suggestion of using a thread to handle coordination with each of the
server processors is the conventional textbook approach; there's
much to recommend doing it the way everyone else does.

I've frequently found it simpler to not deal with thread coordination
problems (which can get real nasty) by using the select or poll
services from the select module (7.3 in the 2.2.3 documentation).
This leads to an event driven style that is closer to GUI programming
than the procedure oriented style typical of threads.

Any way you do it, it's going to complicate your program.

John Roth

> Thanks.
>