Passing functions as parameter (multiprocessing)

[Oscar Benjamin]

On 13 November 2012 12:51, Peter Otten <__peter__ at web.de> wrote:
> Jean-Michel Pichavant wrote:
>
>> I'm having problems understanding an issue with passing function as
>> parameters.
>
>> Here's a code that triggers the issue:
>>
>>
>> import multiprocessing
>>
>> def f1():
>>     print 'I am f1'
>> def f2(foo):
>>     print 'I am f2 %s' % foo
>>
>> workers = [
>>         (f1,tuple()),
>>         (f2,(5,)),
>>         ]
>>
>> procs=[]
>> for func, parameters in workers:
>
>>     def subproc(*args, **kwargs):
>>         return func(*args, **kwargs)
>>     procs.append(multiprocessing.Process(target=subproc, args=parameters))

I don't know if this is to do with the way that the code was
simplified before posting but this subproc function wrapper does
nothing (even after Peter fixed it below). This code is needlessly
complicated for what it does.

> Python has late binding, and when the loop has finished the name func is
> bound to f2. You have created multiple subproc functions, but that doesn't
> matter as they all invoke func aka f2.
>
> A possible fix:
>
> def make_subproc(func):
>     def subproc(*args, **kwargs):
>         return func(*args, **kwargs)
>     return subproc
>
> procs=[]
> for func, parameters in workers:
>     procs.append(multiprocessing.Process(target=make_subproc(func),
> args=parameters))

The above is one way to wrap a function but it wraps the function into
a function that is exactly the same as the original function. I assume
that the reason for having the subproc function was to be able to bind
parameters to the function like so:

def make_subproc(func, args):
    def subproc():
        return func(*args)
    return subproc

procs=[]
for func, parameters in workers:
    procs.append(multiprocessing.Process(target=make_subproc(func, parameters)))

However, as you have already noticed, multiprocess.Process already has
the machinery to do this. If you pass in a value for args you can
leave out the subproc/make_subproc function and write:

procs=[]
for func, parameters in workers:
    procs.append(multiprocessing.Process(target=func, args=parameters))

A loop like this can just be a list comprehension:

procs = [multiprocessing.Process(target=func, args=args) for func,
args in workers]


Oscar