multiprocessing vs thread performance

[Aaron Brady]

On Dec 29, 6:05 pm, "James Mills" <prolo... at shortcircuit.net.au>
wrote:
> On Tue, Dec 30, 2008 at 12:52 AM, mk <mrk... at gmail.com> wrote:
> > Hello everyone,
>
> > After readinghttp://www.python.org/dev/peps/pep-0371/I was under
> > impression that performance of multiprocessing package is similar to that of
> > thread / threading. However, to familiarize myself with both packages I
> > wrote my own test of spawning and returning 100,000 empty threads or
> > processes (while maintaining at most 100 processes / threads active at any
> > one time), respectively.
snip
> As others have mentioned, it's not suprising
> that spawning even 100 processes took some
> time.
>
> Bottom line: multiprocessing should not be used this way.
> (nor should threading).

The OP may be interested in Erlang, which Wikipedia (end-all, be-all)
claims is a 'distribution oriented language'.

You might also find it interesting to examine a theoretical OS that is
optimized for process overhead.  In other words, what is the minimum
overhead possible?  Can processes be as small as threads?  Can entire
threads be only a few bytes (words) big?

Also, could generators provide any of the things you need with your
multiple threads?  You could, say, call 'next()' on many items in a
list, and just remove them on StopIteration.