[SciPy-User] leastsq and multiprocessing

[Frédéric Parrenin]

Answering to my own question:
Actually, the same code runs on debian 7 instead of ubuntu 13.10 does not
slow down my computer. So this may be an ubuntu-specific problem.

For the gain, my program runs in 545 s on one core and in 123 s using 10
cores.
So it seems there is a factor of 2 performance hit in this case, this is
not two bad.

Best regards,

Frédéric Parrenin




2014-05-23 4:23 GMT+02:00 Matt Newville <newville at cars.uchicago.edu>:

> Hi Frederic,
>
> On Thu, May 22, 2014 at 10:20 AM, Frédéric Parrenin <
> parrenin at ujf-grenoble.fr> wrote:
>
>> Dear all,
>>
>> Coming back to an old thread...
>>
>> I tried Jeremy's method since it is the easiest to implement.
>> Below is the Dfun function I provided to leastsq.
>> In my experiment, I used a pool of 6 since I have 8 cores in my PC.
>>
>> However, the computer becomes extremely slow, almost unusable, during the
>> experiment.
>> Do you know why this happens?
>>
>> Best regards,
>>
>> Frédéric
>>
>
> Yes, my observation, based on the code at
>
> https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc
>
> was that there was about a 10x performance hit.    So, similar to your
> observations.
>
> This approach assumes that the cost of setting up multiple processes is
> small compared to the execution time of the objective function itself.  It
> also assumes that having a Jacobian function in Python (as compared to
> Fortran) is a small performance hit.  Again, this is more likely to be true
> for a time-consuming objective function, and almost certainly not true for
> any small test case.
>
> I could be persuaded that this approach is still a reasonable idea, but
> (at least if implemented in pure Python) all the evidence is that it is
> much slower.  Using Cython may help, but I have not tried this.
>
> Any multiprocessing approach that includes calling the objective function
> from different processes is going to be limited by the "picklability"
> issue. To me, this is a fairly significant limitation.  I've been lead to
> believe that the Mystic framework may have worked around this problem, but
> I don't know the details.
>
> Others have suggested that doing the QR factorization with multiprocessing
> would be the better approach.  This seems worth trying, but, In my
> experience, the bulk of the time is actually spent in the objective
> function.
>
> --Matt Newville
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140523/d6c90d5e/attachment.html>