run a function in another processor in python

[Peter Otten]

geremy condra wrote:

> On Fri, Dec 10, 2010 at 4:42 AM, Astan Chee <astan.chee at gmail.com> wrote:
>> On Fri, Dec 10, 2010 at 1:37 AM, Peter Otten <__peter__ at web.de> wrote:
>>>
>>> I can't replicate the crash. However, your problem looks like there is a
>>> ready-to-use solution:
>>>
>>>
>>> http://docs.python.org/library/multiprocessing.html#using-a-pool-of-
workers
>>>
>>> --
>>> http://mail.python.org/mailman/listinfo/python-list
>>
>>
>> Pool.map doesn't seem to be able to support multiple argument functions
>> which is what I'm trying to do here. Any other suggestions?
>> Thanks again
> 
> 1. Post the real code you're using, and
> 2. Put the arguments you want in a tuple and pass that. As an example,
> let's say I have the following function:
> 
> def my_func(x, y, z):
>     return x + y * z
> 
> you could rewrite this as:
> 
> def my_func(*args):
>     return args[0] + args[1] * args[2]
> 
> 
> Here's a worked-out example:
> 
> #! /usr/bin/env python3
> 
> import multiprocessing
> 
> def my_func(x, y, z):
>     return x + y * z
> 
> def my_func_wrapper(t):
>     return my_func(*t)
> 
> # assume we can get an iterable over each argument
> xs = [1, 2, 3, 4]
> ys = [5, 6, 7, 8]
> zs = [9, 1, 2, 3]
> 
> # set up the pool to match the number of CPUs
> num_processes = multiprocessing.cpu_count()
> pool = multiprocessing.Pool(num_processes)
> 
> #will call my_func(1, 5, 9), my_func(2, 6, 1), etc.
> results = pool.map(my_func_wrapper, zip(xs, ys, zs))
> print(results)
> 
> 
> Interesting factoid: you can't seem to use lambda or a decorator to do
> this, which would have been my first instinct. Pickle apparently
> chokes, although marshall wouldn't.

You basically have to ensure that the resulting function is found under its 
__name__ in the global namespace of its __module__. This can be done with 
functools.wraps():

#! /usr/bin/env python3

import multiprocessing
import functools

def starcall(f):
    @functools.wraps(f)
    def g(args):
        return f(*args)
    return g

@starcall
def my_func(x, y, z):
    return x + y * z

xs = [1, 2, 3, 4]
ys = [5, 6, 7, 8]
zs = [9, 1, 2, 3]

num_processes = multiprocessing.cpu_count()
pool = multiprocessing.Pool(num_processes)

results = pool.map(my_func, zip(xs, ys, zs))
print(results)