Multiprocessing, shared memory vs. pickled copies

[Robert Kern]

On 4/4/11 3:20 PM, John Ladasky wrote:
> Hi folks,
>
> I'm developing some custom neural network code.  I'm using Python 2.6,
> Numpy 1.5, and Ubuntu Linux 10.10.  I have an AMD 1090T six-core CPU,
> and I want to take full advantage of it.  I love to hear my CPU fan
> running, and watch my results come back faster.

You will want to ask numpy questions on the numpy mailing list.

   http://www.scipy.org/Mailing_Lists

> When I'm training a neural network, I pass two numpy.ndarray objects
> to a function called evaluate.  One array contains the weights for the
> neural network, and the other array contains the input data.  The
> evaluate function returns an array of output data.
>
> I have been playing with multiprocessing for a while now, and I have
> some familiarity with Pool.  Apparently, arguments passed to a Pool
> subprocess must be able to be pickled.  Pickling is still a pretty
> vague progress to me, but I can see that you have to write custom
> __reduce__ and __setstate__ methods for your objects.  An example of
> code which creates a pickle-friendly ndarray subclass is here:
>
> http://www.mail-archive.com/numpy-discussion@scipy.org/msg02446.html

Note that numpy arrays are already pickle-friendly. This message is telling you 
how, *if* you are already subclassing, how to make your subclass pickle the 
extra information it holds.

> Now, I don't know that I actually HAVE to pass my neural network and
> input data as copies -- they're both READ-ONLY objects for the
> duration of an evaluate function (which can go on for quite a while).
> So, I have also started to investigate shared-memory approaches.  I
> don't know how a shared-memory object is referenced by a subprocess
> yet, but presumably you pass a reference to the object, rather than
> the whole object.   Also, it appears that subprocesses also acquire a
> temporary lock over a shared memory object, and thus one process may
> well spend time waiting for another (individual CPU caches may
> sidestep this problem?) Anyway, an implementation of a shared-memory
> ndarray is here:
>
> https://bitbucket.org/cleemesser/numpy-sharedmem/src/3fa526d11578/shmarray.py
>
> I've added a few lines to this code which allows subclassing the
> shared memory array, which I need (because my neural net objects are
> more than just the array, they also contain meta-data).

Honestly, you should avoid subclassing ndarray just to add metadata. It never 
works well. Make a plain class, and keep the arrays as attributes.

> But I've run
> into some trouble doing the actual sharing part.  The shmarray class
> CANNOT be pickled.

Please never just *say* that something doesn't work. Show us what you tried, and 
show us exactly what output you got. I assume you tried something like this:

[Downloads]$ cat runmp.py
from multiprocessing import Pool
import shmarray


def f(z):
     return z.sum()

y = shmarray.zeros(10)
z = shmarray.ones(10)

p = Pool(2)
print p.map(f, [y, z])


And got output like this:

[Downloads]$ python runmp.py
Exception in thread Thread-2:
Traceback (most recent call last):
   File 
"/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/threading.py", 
line 530, in __bootstrap_inner
     self.run()
   File 
"/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/threading.py", 
line 483, in run
     self.__target(*self.__args, **self.__kwargs)
   File 
"/Library/Frameworks/Python.framework/Versions/7.0/lib/python2.7/multiprocessing/pool.py", 
line 287, in _handle_tasks
     put(task)
PicklingError: Can't pickle <class 
'multiprocessing.sharedctypes.c_double_Array_10'>: attribute lookup 
multiprocessing.sharedctypes.c_double_Array_10 failed


Now, the sharedctypes is supposed to implement shared arrays. Underneath, they 
have some dynamically created types like this c_double_Array_10 type. 
multiprocessing has a custom pickler which has a registry of reduction functions 
for types that do not implement a __reduce_ex__() method. For these dynamically 
created types that cannot be imported from a module, this dynamic registry is 
the only way to do it. At least at one point, the Connection objects which 
communicate between processes would use this custom pickler to serialize objects 
to bytes to transmit them.

However, at least in Python 2.7, multiprocessing seems to have a C extension 
module defining the Connection objects. Unfortunately, it looks like this C 
extension just imports the regular pickler that is not aware of these custom 
types. That's why you get this error. I believe this is a bug in Python.

So what did you try, and what output did you get? What version of Python are you 
using?

> I think that my understanding of multiprocessing
> needs to evolve beyond the use of Pool, but I'm not sure yet.  This
> post suggests as much.
>
> http://mail.scipy.org/pipermail/scipy-user/2009-February/019696.html

Maybe. If the __reduce_ex__() method is implemented properly (and 
multiprocessing bugs aren't getting in the way), you ought to be able to pass 
them to a Pool just fine. You just need to make sure that the shared arrays are 
allocated before the Pool is started. And this only works on UNIX machines. The 
shared memory objects that shmarray uses can only be inherited. I believe that's 
what Sturla was getting at.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco