multiprocessing not quite working

Noah noah-list at enabled.com
Tue May 24 18:17:02 EDT 2016


Hi,

I am using this example: 
http://spartanideas.msu.edu/2014/06/20/an-introduction-to-parallel-programming-using-pythons-multiprocessing-module/

I am sending and receiving communication from the worker processes.

Two issues. the join is only getting to the process and waiting.
When I comment out the .join() process the output.get() appends the 
previous process and therefore the returned output keeps getting longer 
and longer after each process returns its output.

hostnames is an array of hostnames.

here is my code from main():

     # Define an output queue
     output = mp.Queue()

     # Setup a list of processes that we want to run
     processes = [mp.Process(target=worker, args=(hostnames[x], output)) 
for x in range(len(hostnames))]

     # Run processes
     for p in processes:
         print "start: {}".format(p)
         p.start()

     time.sleep(6)
     print "processes: {}".format(processes)

     # Exit the completed processes
     '''for p in processes:
         print "join: {}".format(p)
         p.join()'''

     print "got here"
     # Get process results from the output queue
     # results = [output.get() for p in processes]

     io = StringIO()
     count = 0
     for p in processes:
         found_output = output.get()
         print "returned {}".format(p)
         io.write (found_output)
         zipArchive.writestr(hostnames[count] + "." + 
content['sitename'] + '.config.txt', io.getvalue())
         count = count + 1
     io.close()


def worker(hostname, output):
.
.
.
         output.put(template_output)


Cheers




More information about the Python-list mailing list