Advice regarding multiprocessing module

[Oscar Benjamin]

On 11 March 2013 14:57, Abhinav M Kulkarni <amkulkar at uci.edu> wrote:
> Hi Jean,
>
> Below is the code where I am creating multiple processes:
>
> if __name__ == '__main__':
>     # List all files in the games directory
>     files = list_sgf_files()
>
>     # Read board configurations
>     (intermediateBoards, finalizedBoards) = read_boards(files)
>
>     # Initialize parameters
>     param = Param()
>
>     # Run maxItr iterations of gradient descent
>     for itr in range(maxItr):
>         # Each process analyzes one single data point
>         # They dump their gradient calculations in queue q
>         # Queue in Python is process safe
>         start_time = time.time()
>         q = Queue()
>         jobs = []
>         # Create a process for each game board
>         for i in range(len(files)):
>             p = Process(target=TrainGoCRFIsingGibbs, args=(intermediateBoards[i], finalizedBoards[i], param, q))

Use a multiprocessing.Pool for this, rather than creating one process
for each job. e.g.:

p = Pool(4) # 1 process for each core
results = []
for ib, fb in zip(intermediateBoards, finalizedBoards):
    results.append(p.apply_async(TrainGoCRFIsingGibbs, args=(ib, fb, param, q)))
p.close()
p.join()

# To retrieve the return values
for r in results:
    print(r.get())

This will distribute your jobs over a fixed number of processes. You
avoid the overhead of creating and killing processes and the process
switching that occurs when you have more processes than cores.

>             p.start()
>             jobs.append(p)
>         # Blocking wait for each process to finish
>         for p in jobs:
>             p.join()
>         elapsed_time = time.time() - start_time
>         print 'Iteration: ', itr, '\tElapsed time: ', elapsed_time
>
> As you recommended, I'll use the profiler to see which part of the code is
> slow.

Do this without using multiprocessing first. Loosely you can hope that
multiprocessing would give you a factor of 4 speedup but no more. You
haven't reported a comparison of times with/without multiprocessing so
it's not clear that that is the issue.


Oscar