multiprocessing vs thread performance

[James Mills]

On Tue, Dec 30, 2008 at 12:52 AM, mk <mrkafk at gmail.com> wrote:
> Hello everyone,
>
> After reading http://www.python.org/dev/peps/pep-0371/ I was under
> impression that performance of multiprocessing package is similar to that of
> thread / threading. However, to familiarize myself with both packages I
> wrote my own test of spawning and returning 100,000 empty threads or
> processes (while maintaining at most 100 processes / threads active at any
> one time), respectively.
>
> The results I got are very different from the benchmark quoted in PEP 371.
> On twin Xeon machine the threaded version executed in 5.54 secs, while
> multiprocessing version took over 222 secs to complete!
>
> Am I doing smth wrong in code below? Or do I have to use
> multiprocessing.Pool to get any decent results?

The overhead in starting OS level processes
is quite high. This is why event-driven, single
process servers can perform far better than
ones that fork (spawn multiple processes)
per request.

As others have mentioned, it's not suprising
that spawning even 100 processes took some
time.

Bottom line: multiprocessing should not be used this way.
(nor should threading).

cheers
James