Multiprocessing, shared memory vs. pickled copies

[Dan Stromberg]

On Mon, Apr 4, 2011 at 1:20 PM, John Ladasky <ladasky at my-deja.com> wrote:

> When should one pickle and copy?  When to implement an object in
> shared memory?  Why is pickling apparently such a non-trivial process
> anyway?  And, given that multi-core CPU's are apparently here to stay,
> should it be so difficult to make use of them?<http://mail.python.org/mailman/listinfo/python-list>


Pickle and copy when your distinct processes will benefit from  having
multiple distinct copies of the data - that is, copies that can diverge
from  each other if written to in one or more of the processes.

Use shared memory if your distinct processes need to be able to see a single
copy of the same data - EG, because one needs to see the result of the
changes made in another.

Pickling's not a big deal -  it's just turning structured data (from an int
to a nested series of objects) into a stream of bytes.  Not all extension
types support it without extra work though.

Yes, multicore is increasingly  important, and Python needs to support it
well.  multiprocessing is one good way of doing so.  Also, Python 3.2 has a
new module facilitating this,  and an improved GIL situation.  Then there
are things like greenlets and stackless python.  Oh, and if you move to
jython or ironpython, you get better multithreading.

Multithreading is generally a bit lighter-weight than multiprocessing, but
it also gives a little tighter coupling between different parts of the
program and a greater probability of race conditions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110404/ee68f9f8/attachment-0001.html>