Running a python farm

[Hung Jung Lu]

Ian McConnell <ian at emit.demon.co.ukx> wrote in message news:<877k2qrc0a.fsf at emit.demon.co.uk>...
> Processing the image can take several minutes and I have loads of images to
> process, but I also have several servers available, so I'd like save some
> time by distributing the images around the servers. So for 'n' servers,
> throw 'n' images at them. Then as each server finishes, give it another
> image to work on until I've done all the images.

That's all fine. In the short term, that's the way to get things done.
You many also want to look at xmlrpclib.

For longer term, you may want to do extra decoupling, to make things
more scalable/maintainable/fault-tolerant. A few ideas I got recently
were: (1) prepare tasks well enough so they can kind of run "off-line"
in a pool, you may use GUIDs (globally unique ids) to tag the tasks
(2) do remote-to-local calls instead of local-to-remote calls. This
means you need broadcast capability. (3) Write your own task manager,
an installable on all machines (local and remote), so that you can
dispatch programs and data files more easily, start and stop remote
services. For instance, you now may have a remote server to do image
processing, but what in the future you want to have servers to do
sound file processing? What if you need to upgrade your remote
programs?

regards,

Hung Jung