Persistent variable in subprocess using multiprocessing?

[Piet van Oostrum]

>>>>> mheavner <miheavner at gmail.com> (m) wrote:

>m> 'The process' refers to the subprocess. I could do as you say, load
>m> the data structure each time, but the problem is that takes a
>m> considerable amount of time compared to the the actual computation
>m> with the data it contains. I'm using these processes within a loop as
>m> follows:

>m>          # Don't recreate processes or Queues
>m>          pop1 = Queue()
>m>          pop2 = Queue()
>m>          pop_out = Queue()
>m>          p1 = CudaProcess(0, args=(costf,pop1,pop_out))
>m>          p2 = CudaProcess(1, args=(costf,pop2,pop_out))

>m>          # Main loop
>m>          for i in range(maxiter):
>m>                  print 'ITERATION: '+str(i)
>m>                  if log != None:
>m>                          l = open(log,'a')
>m>                  l.write('Iteration: '+str(i)+'\n')
>m>                  l.close()

>m>                  # Split population in two
>m>                  pop1.putmany(pop[0:len(pop)/2])
>m>                  pop2.putmany(pop[len(pop)/2:len(pop)])

>m>                  # Start two processes
>m>                  if not p1.isAlive():
>m>                          p1.start()
>m>                          print 'started %s'%str(p1.getPid())
>m>                  else:
>m>                          p1.run()

That won't work. p1.run() will execute the run method in the Master
process, not in the subprocess. And if it would your could would have a
race condition: between the p1.isAlive() (which must be is_alive btw), and
the p1.run() the process can have stopped.

The proper way to do is to put the work in a Queue and let the processes
get work out of the Queue. The datastructure will remain in the process
then. 

>m>                  if not p2.isAlive():
>m>                          p2.start()
>m>                          print 'started %s'%str(p2.getPid())
>m>                  else:
>m>                          p2.run()
>m>                  .
>m>                  .
>m>                  .

>m> So I'd like to load that data into memory once and keep there as long
>m> as the process is alive (ideally when the subprocess is created,
>m> storing some sort of pointer to it), rather than loading it each time
>m> run is called for a process within the loop. Could be my CudaProcess
>m> class - I'll check out what Diez suggested and post back.

-- 
Piet van Oostrum <piet at cs.uu.nl>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: piet at vanoostrum.org