Multiprocessing, shared memory vs. pickled copies

[Philip Semanchuk]

On Apr 5, 2011, at 12:58 PM, John Ladasky wrote:

> Hi Philip,
> 
> Thanks for the reply.
> 
> On Apr 4, 4:34 pm, Philip Semanchuk <phi... at semanchuk.com> wrote:
>> So if you're going to use multiprocessing, you're going to use pickle, and you
>> need pickleable objects.
> 
> OK, that's good to know.

But as Dan Stromberg pointed out, there are some pickle-free ways to communicate between processes using multiprocessing.

> This leads straight into my second question.  I THINK, without knowing
> for sure, that most user classes would pickle correctly by simply
> iterating through __dict__.  So, why isn't this the default behavior
> for Python?  Was the assumption that programmers would only want to
> pickle built-in classes?  

One can pickle user-defined classes:

>>> class Foo(object):
...     pass
... 
>>> import pickle
>>> foo_instance = Foo()
>>> pickle.dumps(foo_instance)
'ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'


And as Robert Kern pointed out, numpy arrays are also pickle-able.

>>> import numpy
>>> pickle.dumps(numpy.zeros(3))
"cnumpy.core.multiarray\n_reconstruct\np0\n(cnumpy\nndarray\np1\n(I0\ntp2\nS'b'\np3\ntp4\nRp5\n(I1\n(I3\ntp6\ncnumpy\ndtype\np7\n(S'f8'\np8\nI0\nI1\ntp9\nRp10\n(I3\nS'<'\np11\nNNNI-1\nI-1\nI0\ntp12\nbI00\nS'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\np13\ntp14\nb."

As a side note, you should always use "new style" classes, particularly since you're exploring the details of Python class construction. "New" is a bit a of misnomer now, as "new" style classes were introduced in Python 2.2. They have been the status quo in Python 2.x for a while now and are the only choice in Python 3.x.

Subclassing object gives you a new style class:
   class Foo(object):

Not subclassing object (as you did in your example) gives you an old style class:
   class Foo:



Cheers
Philip