Using Pool map with a method of a class and a list

[Peter Otten]

Joshua Landau wrote:

> On 7 August 2013 11:10, Luca Cerone <luca.cerone at gmail.com> wrote:
>> I can't try it now, I'll let you know later if it works!
>> (Though just by reading I can't really understand what the code does).
> 
> Well,
> 
>>> from multiprocessing import Pool
>>> from functools import partial
>>>
>>> class A(object):
>>>     def __init__(self,x):
>>>         self.value = x
>>>     def fun(self,x):
>>>         return self.value**x
> 
> This is all the same, as with
> 
>>> l = range(100)
>>> p = Pool(4)
> 
> You then wanted to do:
> 
>> op = p.map(A(3).fun, l)
> 
> but bound methods can't be pickled, it seems.
> 
> However, A(3) *can* be pickled. So what we want is a function:
> 
>     def proxy(arg):
>         A(3).fun(arg)
> 
> so we can write:
> 
>> op = p.map(proxy, l)
> 
> To generalise you might be tempted to write:
> 
>     def generic_proxy(instance, name):
>         def proxy(arg):
>             # Equiv. of instance.name(arg)
>             getattr(instance, name)(arg)
> 
> but the inner function won't work as functions-in-functions can't be
> pickled either.
> 
> So we use:
> 
>>> def _getattr_proxy_partialable(instance, name, arg):
>>>     return getattr(instance, name)(arg)
> 
> Which takes all instance, name and arg. Of course we only want our
> function to take arg, so we partial it:
> 
>>> def getattr_proxy(instance, name):
>>>     """
>>>     A version of getattr that returns a proxy function that can
>>>     be pickled. Only function calls will work on the proxy.
>>>     """
>>>     return partial(_getattr_proxy_partialable, instance, name)
> 
> partial objects are picklable, btw.
> 
>>> op = p.map(getattr_proxy(A(3), "fun"), l)
>>> print(op)
> 
> :)


There is also the copy_reg module. Adapting

<http://mail.python.org/pipermail/python-list/2008-July/469164.html>

you get:

import copy_reg
import multiprocessing
import new

def make_instancemethod(inst, methodname):
    return getattr(inst, methodname)

def pickle_instancemethod(method):
    return make_instancemethod, (method.im_self, method.im_func.__name__)

copy_reg.pickle(
    new.instancemethod, pickle_instancemethod, make_instancemethod)

class A(object):
   def __init__(self, a):
       self.a = a
   def fun(self, b):
       return self.a**b

if __name__ == "__main__":
    items = range(10)
    pool = multiprocessing.Pool(4)
    print pool.map(A(3).fun, items)