examples of realistic multiprocessing usage?

[Dan Stromberg]

On Sun, Jan 16, 2011 at 11:05 AM, TomF <tomf.sessile at gmail.com> wrote:
> I'm trying to multiprocess my python code to take advantage of multiple
> cores.  I've read the module docs for threading and multiprocessing, and
> I've done some web searches.  All the examples I've found are too simple:
> the processes take simple inputs and compute a simple value.  My problem
> involves lots of processes, complex data structures, and potentially lots of
> results.  It doesn't map cleanly into a Queue, Pool, Manager or
> Listener/Client example from the python docs.
>
> Instead of explaining my problem and asking for design suggestions, I'll
> ask: is there a compendium of realistic Python multiprocessing examples
> somewhere?  Or an open source project to look at?

I'm unaware of a big archive of projects that use multiprocessing, but
maybe one of the free code search engines could help with that.

It sounds like you're planning to use mutable shared state, which is
generally best avoided if at all possible, in concurrent programming -
because mutable shared state tends to slow down things quite a bit.

But if you must have mutable shared state that's more complex than a
basic scalar or homogeneous array, I believe the multiprocessing
module would have you use a "server process manager".