Using Pool map with a method of a class and a list

[Joshua Landau]

On 7 August 2013 11:10, Luca Cerone <luca.cerone at gmail.com> wrote:
> I can't try it now, I'll let you know later if it works!
> (Though just by reading I can't really understand what the code does).

Well,

>> from multiprocessing import Pool
>> from functools import partial
>>
>> class A(object):
>>     def __init__(self,x):
>>         self.value = x
>>     def fun(self,x):
>>         return self.value**x

This is all the same, as with

>> l = range(100)
>> p = Pool(4)

You then wanted to do:

> op = p.map(A(3).fun, l)

but bound methods can't be pickled, it seems.

However, A(3) *can* be pickled. So what we want is a function:

    def proxy(arg):
        A(3).fun(arg)

so we can write:

> op = p.map(proxy, l)

To generalise you might be tempted to write:

    def generic_proxy(instance, name):
        def proxy(arg):
            # Equiv. of instance.name(arg)
            getattr(instance, name)(arg)

but the inner function won't work as functions-in-functions can't be
pickled either.

So we use:

>> def _getattr_proxy_partialable(instance, name, arg):
>>     return getattr(instance, name)(arg)

Which takes all instance, name and arg. Of course we only want our
function to take arg, so we partial it:

>> def getattr_proxy(instance, name):
>>     """
>>     A version of getattr that returns a proxy function that can
>>     be pickled. Only function calls will work on the proxy.
>>     """
>>     return partial(_getattr_proxy_partialable, instance, name)

partial objects are picklable, btw.

>> op = p.map(getattr_proxy(A(3), "fun"), l)
>> print(op)

:)