multiprocessing vs thread performance

[Aaron Brady]

On Dec 29, 9:29 am, mk <mrk... at gmail.com> wrote:
> Christian Heimes wrote:
> > mk wrote:
> >> Am I doing smth wrong in code below? Or do I have to use
> >> multiprocessing.Pool to get any decent results?
>
> > You have missed an important point. A well designed application does
> > neither create so many threads nor processes.
>
> Except I was not developing "well designed application" but writing the
> test the goal of which was measuring the thread / process creation cost.
>
> > The creation of a thread
> > or forking of a process is an expensive operation.
>
> Sure. The point is, how expensive? While still being relatively
> expensive, it turns out that in Python creating a thread is much, much
> cheaper than creating a process via multiprocessing on Linux, while this
> seems to be not necessarily true on Mac OS X.
>
> > You should use a pool
> > of threads or processes.
>
> Probably true, except, again, that was not quite the point of this
> exercise..
>
> > The limiting factor is not the creation time but the communication and
> > synchronization overhead between multiple threads or processes.
>
> Which I am probably going to test as well.

I had an idea.  You could use 'multiprocessing' for synchronization,
and just use an mmap for the actual communication.  (I think it's a
good idea.)