client-server parallellised number crunching

[Chris Angelico]

On Wed, Apr 27, 2011 at 5:55 AM, Hans Georg Schaathun
<georg at schaathun.net> wrote:
> It is, of course, possible for the master thread upon processing the
> results, to requeue the tasks for any missing results, but it seems
> to me to be a cleaner solution if I could detect disconnects and
> requeue the tasks from the networking threads.  Is that possible
> using python sockets?
>
> Somebody will probably ask why I am not using one of the multiprocessing
> libraries.  I have tried at least two, and got trapped by the overhead
> of passing complex pickled objects across.

If I were doing this, I would devise my own socket-layer protocol and
not bother with pickling objects at all. The two ends would read and
write the byte stream and interpret it as data.

But question: Why are you doing major number crunching in Python? On
your quad-core machine, recode in C and see if you can do the whole
job without bothering the unreliable boxen at all.

Chris Angelico