[Python-Dev] multiprocessing not compatible with functional.partial

[Neal Becker]

Hrvoje Niksic wrote:

> Calvin Spealman wrote:
>> I don't think it would be unreasonable to consider either 1) making
>> functools.partial picklable (I don't know how feasible this is)
> 
> It's not only feasible, but quite easy and, I think, useful.  A
> "partial" instance is a simple triplet of (function, args, kwds), and it
> can be pickled as such.  For example:
> 
>  >>> import copy_reg, functools
>  >>> def _reconstruct_partial(f, args, kwds):
> ...     return functools.partial(f, *args, **(kwds or {}))
> ...
>  >>> def _reduce_partial(p):
> ...     return _reconstruct_partial, (p.func, p.args, p.keywords)
> ...
>  >>> copy_reg.pickle(functools.partial, _reduce_partial)
> 
> Test:
> 
>  >>> import operator, cPickle as cp
>  >>> p = functools.partial(operator.add, 3)
>  >>> p(10)
> 13
>  >>> cp.dumps(p)
> 
'c__main__\n_reconstruct_partial\np1\n(coperator\nadd\np2\n(I3\ntp3\nNtRp4\n.'
>  >>> p2 = cp.loads(_)
>  >>> p2(10)
> 13
> 
> Iedally this should be implemented in the functools.partial object itself.
Confirmed:

from multiprocessing import Pool

def power (x, pwr=2):
    return x**pwr

import functools
run_test = functools.partial (power, pwr=3)

import copy_reg, functools
def _reconstruct_partial(f, args, kwds):
    return functools.partial(f, *args, **(kwds or {}))

def _reduce_partial(p):
    return _reconstruct_partial, (p.func, p.args, p.keywords)

copy_reg.pickle(functools.partial, _reduce_partial)

if __name__ == "__main__":

    pool = Pool()
    cases = [3,4,5]
    results = pool.map (run_test, cases)
    print results

$python test_multi.py
[27, 64, 125]