examples of realistic multiprocessing usage?

[Philip Semanchuk]

On Jan 16, 2011, at 2:05 PM, TomF wrote:

> I'm trying to multiprocess my python code to take advantage of multiple cores.  I've read the module docs for threading and multiprocessing, and I've done some web searches.  All the examples I've found are too simple: the processes take simple inputs and compute a simple value.  My problem involves lots of processes, complex data structures, and potentially lots of results.  It doesn't map cleanly into a Queue, Pool, Manager or Listener/Client example from the python docs.
> 
> Instead of explaining my problem and asking for design suggestions, I'll ask: is there a compendium of realistic Python multiprocessing examples somewhere?  Or an open source project to look at?


A colleague pointed me to this project the other day. 

http://gluino.com/


I grepped through the code to see that it's using multiprocessing.Listener. I didn't go any further than that because our project is BSD licensed and the license for Gluino is unclear. Until I find out whether or not its under an equally permissive license, I can't borrow ideas and/or code from it.

Hope it's of some help to you, though.

Cheers
Philip