multiprocessing vs thread performance

[Gabriel Genellina]

En Wed, 07 Jan 2009 23:05:53 -0200, James Mills  
<prologic at shortcircuit.net.au> escribió:

>> Does anybody know any tutorial for python 2.6 multiprocessing? Or bunch  
>> of
>> good example for it? I am trying to break a loop to run it over multiple
>> core in a system. And I need to return an integer value as the result  
>> of the
>> process an accumulate all of them. the examples that I found there is no
>> return for the process.
>
> You communicate with the process in one of several
> ways:
>  * Semaphores
>  * Locks
>  * PIpes

The Pool class provides a more abstract view that may be better suited in  
this case. Just create a pool, and use map_async to collect and summarize  
the results.

import string
import multiprocessing

def count(args):
     (lineno, line) = args
     print "This is %s, processing line %d\n" % (
         multiprocessing.current_process().name, lineno),
     result = dict(letters=0, digits=0, other=0)
     for c in line:
         if c in string.letters: result['letters'] += 1
         elif c in string.digits: result['digits'] += 1
         else: result['other'] += 1
     # just to make some "random" delay
     import time; time.sleep(len(line)/100.0)
     return result

if __name__ == '__main__':

     summary = dict(letters=0, digits=0, other=0)

     def summary_add(results):
         # this is called with a list of results
         for result in results:
             summary['letters'] += result['letters']
             summary['digits'] += result['digits']
             summary['other'] += result['other']

     # count letters on this same script
     f = open(__file__, 'r')

     pool = multiprocessing.Pool(processes=6)
     # invoke count((lineno, line)) for each line in the file
     pool.map_async(count, enumerate(f), 10, summary_add)
     pool.close() # no more jobs
     pool.join()  # wait until done
     print summary

-- 
Gabriel Genellina