run a function in another processor in python

[Astan Chee]

Sorry about that, here is a summary of my complete code. I haven't cleaned
it up much or anything, but this is what it does:

import time
import multiprocessing

test_constx =0
test_consty =0

def functionTester(x):
  global test_constx
  global test_consty
  print "constx " + str(test_constx)
  print "consty " + str(test_consty)
  return (test_constx*x[0]-x[1]+test_consty*x[0]+x[2])

def functionTesterMain(constx,consty):
  global test_constx
  global test_consty
  test_constx = constx
  test_consty = consty
  num_args =
[(61,12,1),(61,12,2),(61,12,3),(61,11,4),(61,12,4),(62,33,4),(7,12,4),(16,19,4),(35,36,4),(37,38,3),(55,56,3),(57,63,3)]
  num_processes = multiprocessing.cpu_count()
  pool = multiprocessing.Pool(num_processes)
  rs = []
  start = time.time()
  rs = pool.map(functionTester,num_args)
  end = time.time()
  elapsed= end - start
  min = elapsed/60
  print "Took", elapsed, "seconds to run, which is the same as", min,
"minutes"
  pos = 0
  high = 0
  n = None
  for r in rs:
    if r > high:
      n = num_args[pos]
      high = r
    pos+=1
  print "high " + str(high)
  print "n " + str(n)

  return high,n

if __name__ == '__main__':
  for i in range(1,4):
    a,b = functionTesterMain(i,7)
    print "-----------"
    print "a " + str(a)
    print "b " + str(a)


  Which doesn't seem to work because the functionTester() needs to be
simpler and not use global variables.
I'm using global variables because I'm also trying to pass a few other
variables and I tried using a class but that just gave me a unpickleable
error. I tried using zip but I'm confused with how I can get to pass the
data.
I know I can probably combine the data into tuples but that means that there
is alot of data duplication, especially if the constx and consty are large
dictionaries (or even custom objects), which might happen later.
So it seems that map doesn't quite like functions like these. Anyway, I'll
try and see if threads or something can substitute. I'd appriciate any help.
Thanks



On Sat, Dec 11, 2010 at 9:04 AM, geremy condra <debatem1 at gmail.com> wrote:

>  On Fri, Dec 10, 2010 at 4:42 AM, Astan Chee <astan.chee at gmail.com> wrote:
> > On Fri, Dec 10, 2010 at 1:37 AM, Peter Otten <__peter__ at web.de> wrote:
> >>
> >> I can't replicate the crash. However, your problem looks like there is a
> >> ready-to-use solution:
> >>
> >>
> >>
> http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers
> >>
> >> --
> >> http://mail.python.org/mailman/listinfo/python-list
> >
> >
> > Pool.map doesn't seem to be able to support multiple argument functions
> > which is what I'm trying to do here. Any other suggestions?
> > Thanks again
>
> 1. Post the real code you're using, and
> 2. Put the arguments you want in a tuple and pass that. As an example,
> let's say I have the following function:
>
> def my_func(x, y, z):
>    return x + y * z
>
> you could rewrite this as:
>
> def my_func(*args):
>    return args[0] + args[1] * args[2]
>
>
> Here's a worked-out example:
>
> #! /usr/bin/env python3
>
> import multiprocessing
>
> def my_func(x, y, z):
>    return x + y * z
>
> def my_func_wrapper(t):
>    return my_func(*t)
>
> # assume we can get an iterable over each argument
> xs = [1, 2, 3, 4]
> ys = [5, 6, 7, 8]
> zs = [9, 1, 2, 3]
>
> # set up the pool to match the number of CPUs
> num_processes = multiprocessing.cpu_count()
> pool = multiprocessing.Pool(num_processes)
>
> #will call my_func(1, 5, 9), my_func(2, 6, 1), etc.
> results = pool.map(my_func_wrapper, zip(xs, ys, zs))
> print(results)
>
>
> Interesting factoid: you can't seem to use lambda or a decorator to do
> this, which would have been my first instinct. Pickle apparently
> chokes, although marshall wouldn't.
>
> Geremy Condra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20101211/286d046d/attachment-0001.html>