[Python-Dev] multiprocessing vs. distributed processing

[Matthieu Brucher]

(Sorry for the double send...)

2009/1/16 James Mills <prologic at shortcircuit.net.au>:
> I've noticed over the past few weeks lots of questions
> asked about multi-processing (including myself).

Funny, I was going to blog about this, but not just for Python.

> For those of you new to multi-processing, perhaps this
> thread may help you. Some things I want to start off
> with to point out are:
>
> "multiprocessing will not always help you get things done faster."

Of course. There are some programs that are I/O or memory bandwidth
bound. So if one of those bottlenecks is common to the cores you use,
you can't benefit from their use.

> "be aware of I/O bound applications vs. CPU bound"

Exactly. We read a lot about Folding at Home, SETI at Home, they can be
distributed, as it is more or less "take a chunk, process it somewhere
and when you're finish tell me if there something interesting in it".
Not a lot of communications between the nodes. Then, there are other
applications that process a lot of data, they must read data from
memory, make one computation, read other data, compute a little bit
(finite difference schemes), and here we are memory bandwidth bound,
not CPU bound.

> "multiple CPUs (cores) can compute multiple concurrent expressions -
> not read 2 files concurrently"

Let's say that this is true for the usual computers. Clusters can make
concurrent reads, as long as there is the correct architecture behind.
Of course, if you only have one hard disk, you are limited.

> "in some cases, you may be after distributed processing rather than
> multi or parallel processing"

Of course. Clusters can be expensive, their interconnections even
more. So if your application is made of independent blocks that can
run on small nodes, without much I/Os, you can try distributed
computing. If you need big nodes with high-speed interconnections, you
will have to use parallel processing.

This is just what my thoughts on the sucjet are, but I think I'm not
far from the truth. Of course, if I'm proved wrong, I'll be glad to
hear why.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher