Advice regarding multiprocessing module

[Dave Angel]

On 03/11/2013 01:57 AM, Abhinav M Kulkarni wrote:
>
>
>   <SNIP>
>   * My laptop has quad-core Intel i5 processor, so I thought using
>     multiprocessing module I can parallelize my code (basically
>     calculate gradient in parallel on multiple cores simultaneously).
    <SNIP>
>   * As a result I end up creating a process for each data point (instead
>     of a thread that I would ideally like to do, so as to avoid process
>     creation overhead).

Seems you only need 4 processes, as you have 4 cores.  Instead of 
creating a new one each time, reuse the same 4 processes, letting each 
do a quarter of the data.

It's not the process creation that's particularly slow, but all the 
initialization of starting another instance of Python.  If you're on 
Linux, you might be able to speed that up by using fork, but I don't 
specifically know.



-- 
DaveA