client-server parallellised number crunching

[Hans Georg Schaathun]

On Tue, 26 Apr 2011 23:54:24 -0700, geremy condra
  <debatem1 at gmail.com> wrote:
:  This sounds like a hadoop job, with the caveat that you still have to
:  get your objects across the network somehow. Have you tried xdrlib or
:  the struct module? I suspect either would save you some time.

Packing the objects up does not appear as a problem at the moment.
Pickled objects work just fine as long as I stick to relatively
simple data types and avoid any redundant information from my own
data structure.  When the CPU-s run a about 85% with a naive com's
format, I am not going to waste any time to speed up com's. 

The problem would occur when a client holding a task crashes.

:  Hmm. I guess I'd boil it down to this: if you have the ability to
:  install software on them, give hadoop a try. If, OTOH you can't
:  disturb normal lab operation at all and need a lot of CPU power, you
:  should probably start weighing what your time is worth against the
:  cost of firing up a few EC2 instances and being done with it- I use it
:  for cryptanalytic work with a similar structure all the time, and you
:  can get a hell of a cluster going over there for about $15-20 an hour.

That is an interesting thought for future work, but with the trial
and error I do at the moment, the cost of EC2 would be quite
significant, especially when I have so many idle CPU-s at my finger tips.

:  If you're an academic (it sounds like you are) you may also be able to
:  use things like planetlab and emulab, which are free and reasonably
:  easy to use.

I am.  

Based on other answers, I should be able to hack together the necessary
error checking to handle lost clients with my current solution using
the standard socket and threading API-s.  With the added benefit that
the familiarity with sockets and threads will be useful for teaching in 
the foreseeable future.

It seems that all the tools you suggest will require much more reading
with less potential for reusing the knowledge.  It will be very useful
when I can commit more resources for a longer period of time, and 
certainly worth remembering for any fully-funded project in the future.

-- 
:-- Hans Georg