[SciPy-User] leastsq and multiprocessing

[Frédéric Parrenin]

Actually, the parallel leastsq code is very unstable on both debian 7 and
ubuntu 13.10.
Sometimes it works, sometimes it freezes my computer.

I would be glad if anybody could explain to me the origin of this problem.

Best regards,

Frédéric Parrenin




2014-05-23 8:52 GMT+02:00 Frédéric Parrenin <parrenin at ujf-grenoble.fr>:

> Answering to my own question:
> Actually, the same code runs on debian 7 instead of ubuntu 13.10 does not
> slow down my computer. So this may be an ubuntu-specific problem.
>
> For the gain, my program runs in 545 s on one core and in 123 s using 10
> cores.
> So it seems there is a factor of 2 performance hit in this case, this is
> not two bad.
>
> Best regards,
>
> Frédéric Parrenin
>
>
>
>
> 2014-05-23 4:23 GMT+02:00 Matt Newville <newville at cars.uchicago.edu>:
>
>> Hi Frederic,
>>
>> On Thu, May 22, 2014 at 10:20 AM, Frédéric Parrenin <
>> parrenin at ujf-grenoble.fr> wrote:
>>
>>> Dear all,
>>>
>>> Coming back to an old thread...
>>>
>>> I tried Jeremy's method since it is the easiest to implement.
>>> Below is the Dfun function I provided to leastsq.
>>> In my experiment, I used a pool of 6 since I have 8 cores in my PC.
>>>
>>> However, the computer becomes extremely slow, almost unusable, during
>>> the experiment.
>>> Do you know why this happens?
>>>
>>> Best regards,
>>>
>>> Frédéric
>>>
>>
>> Yes, my observation, based on the code at
>>
>> https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc
>>
>> was that there was about a 10x performance hit.    So, similar to your
>> observations.
>>
>> This approach assumes that the cost of setting up multiple processes is
>> small compared to the execution time of the objective function itself.  It
>> also assumes that having a Jacobian function in Python (as compared to
>> Fortran) is a small performance hit.  Again, this is more likely to be true
>> for a time-consuming objective function, and almost certainly not true for
>> any small test case.
>>
>> I could be persuaded that this approach is still a reasonable idea, but
>> (at least if implemented in pure Python) all the evidence is that it is
>> much slower.  Using Cython may help, but I have not tried this.
>>
>> Any multiprocessing approach that includes calling the objective function
>> from different processes is going to be limited by the "picklability"
>> issue. To me, this is a fairly significant limitation.  I've been lead to
>> believe that the Mystic framework may have worked around this problem, but
>> I don't know the details.
>>
>> Others have suggested that doing the QR factorization with
>> multiprocessing would be the better approach.  This seems worth trying,
>> but, In my experience, the bulk of the time is actually spent in the
>> objective function.
>>
>> --Matt Newville
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140526/1ab0a280/attachment.html>