Using Pool map with a method of a class and a list

[Joshua Landau]

On 7 August 2013 09:33, Luca Cerone <luca.cerone at gmail.com> wrote:
> To correct my example:
>
> from multiprocessing import Pool
>
> class A(object):
>     def __init__(self,x):
>         self.value = x
>     def fun(self,x):
>         return self.value**x
>
> l = range(100)
> p = Pool(4)
> op = p.map(A(3).fun, l)
>
> doesn't work neither in Python 2.7, nor 3.2 (by the way I can't use Python 3 for my application).

Are you using Windows? Over here on 3.3 on Linux it does. Not on 2.7 though.

>> You will find that
>> http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-> > when-using-pythons-multiprocessing-pool-ma
>> explains the problem in more detail than I understand. I suggest
>> reading it and relaying further questions back to us. Or use Python 3
>
> :) Thanks, but of course I googled and found this link before posting. I don't understand much of the details as well, that's why I posted here.
>
> Anyway, thanks for the attempt :)

Reading there, the simplest method seems to be, in effect:

from multiprocessing import Pool
from functools import partial

class A(object):
    def __init__(self,x):
        self.value = x
    def fun(self,x):
        return self.value**x

def _getattr_proxy_partialable(instance, name, arg):
    return getattr(instance, name)(arg)

def getattr_proxy(instance, name):
    """
    A version of getattr that returns a proxy function that can
    be pickled. Only function calls will work on the proxy.
    """
    return partial(_getattr_proxy_partialable, instance, name)

l = range(100)
p = Pool(4)
op = p.map(getattr_proxy(A(3), "fun"), l)
print(op)