Parallel Processing

[David Hoese]

On 1/8/12 1:45 PM, Yigit Turgut <y.turgut at gmail.com> wrote:
> There are no imports other than defined on the script, which are;
>
> import pygame
> import sys
> import time
> import math
> import pp
>
> You are correct about  trying to pass two functions and second one is
> in place where a tuple of arguments supposed to be. But what if these
> functions don't have any arguments ? I tested functions test1() and
> test2() seperately ; they work. Once I figure out how to run these
> functions simultaneously, I will add an argument to test2 and try then
> on. My main goal is to simultaneously run two functions, one of them
> has one argument the other doesn't. To get familiar with parallel
> processing I am experimenting now without arguments and then I will
> embed the code to my application. I am experimenting with the
> following ;
>
> import pygame
> import sys
> import time
> import math
> import pp
>
> screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
> timer = pygame.time.Clock()
> white = True
> start = time.time()
> end = time.time() - start
>
> def test1():
>    global end
>    global white
>    while(end<5):
>      end = time.time() - start
>      timer.tick(4) #FPS
>      screen.fill((255,255,255) if white else (0, 0, 0))
>      white = not white
>      pygame.display.update()
>
> def test2():
>    global end
>    while(end<5):
>      end = time.time() - start
>      print end
>
> ppservers = ()
> job_server = pp.Server(ppservers=ppservers)
> print "Starting pp with", job_server.get_ncpus(), "workers"
>
> job1 = job_server.submit(test1())
> job2 = job_server.submit(test2())
> result = job1()
> result2 = job2()
>
> print "Counting...", result2
>
> job_server.print_stats()
>
> test1() works as expected (job1) but test2() doesn't work and I get
> the following traceback ;
>
> Traceback (most recent call last):
>    File "fl.py", line 33, in<module>
>      job1 = job_server.submit(test1())
>    File "/usr/lib/python2.6/site-packages/pp.py", line 458, in submit
>      sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
>    File "/usr/lib/python2.6/site-packages/pp.py", line 629, in
> __dumpsfunc
>      sources = [self.__get_source(func) for func in funcs]
>    File "/usr/lib/python2.6/site-packages/pp.py", line 696, in
> __get_source
>      sourcelines = inspect.getsourcelines(func)[0]
>    File "/usr/lib/python2.6/inspect.py", line 678, in getsourcelines
>      lines, lnum = findsource(object)
>    File "/usr/lib/python2.6/inspect.py", line 519, in findsource
>      file = getsourcefile(object) or getfile(object)
>    File "/usr/lib/python2.6/inspect.py", line 441, in getsourcefile
>      filename = getfile(object)
>    File "/usr/lib/python2.6/inspect.py", line 418, in getfile
>      raise TypeError('arg is not a module, class, method, '
> TypeError: arg is not a module, class, method, function, traceback,
> frame, or code object
>
> Error is related to test1 not having an argument.  When I leave it
> empty as following ;
>
> job1 = job_server.submit(test1,())
>
> test1 doesn't run. When I do ;
>
> job1 = job_server.submit(test1())
>
> Display works but I get;
>
> TypeError: arg is not a module, class, method, function, traceback,
> frame, or code object (complete traceback same as above).
>
> And test2 doesn't work also. But when I do;
>
> job1 = job_server.submit(test1,())
> job2 = job_server.submit(test2())
>
> I get test2 working but test1 not working. Obviously related to
> argument arrangement in submit.
Hi,

I've never used pygame or Parallel Python, but I played around with the 
code you provided and did one of my favorite debugging techniques...I 
printed things out and read the output.

So one thing I did was printed the globals before the function 
definitions and inside test1().  Which the first print shows what I 
expect from calling "print globals()", then inside test1() I only get 
functions, modules, and a few other things.  So I checked the pp 
documentation and found this about the globals keyword:

     globals - dictionary from which all modules, functions and classes

It also handles imports funny because I tried doing "from pprint import 
pprint" and it couldn't find it properly even though that's a function 
(it couldn't find a class that that function uses).  So I think you'll 
have to pass things in as arguments or a dependency functions as others 
have suggested.  There is also a 'modules' keyword that you can provide 
names of modules to import, which might help.  And is there a reason you 
need to use Parallel Python and can't use something more simple like 
python's "multiprocessing" or the classic "os.fork()"?  I understand 
that Parallel Python can run on remote servers in parallel...but how 
complicated is your program going to be?

I got the following to work (not sure if its what you want):
###
import pp

def test1():
     start = time.time()
     end = time.time() - start
     screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
     timer = pygame.time.Clock()
     white = True
     while(end<5):
         end = time.time() - start
     timer.tick(4) #FPS
     screen.fill((255,255,255) if white else (0, 0, 0))
     white = not white
     pygame.display.update()

def test2():
     start = time.time()
     end2= time.time() - start
     while(end2<5):
         end2 = time.time() - start
     print end2

ppservers = ()
job_server = pp.Server(ppservers=ppservers)

job1 = job_server.submit(test1, modules=("pygame","time"))
job2 = job_server.submit(test2, modules=("time",))
result = job1()
result2 = job2()

print  result2

job_server.print_stats()
###

However, I don't know if this will always work the way you want it to, 
depending on how you setup your Parallel Python servers.  By that I 
mean, if you run this on any machine that isn't local, I think it will 
try to connect to that remote display when getting the pygame "screen".  
But again, I've never used pygame.  And this was also a quick throw 
together, so you could probably pass in "start" and stuff like that so 
that it doesn't have to be calculated both times.

Summary: Parallel Python doesn't handle global variables in a normal way 
(it doesn't like things that aren't functions, modules, or classes) 
so...don't use globals.  Let me know if any of that didn't make sense.

-Dave

P.S. If anyone has any other results I would be curious to hear.