[SciPy-User] leastsq and multiprocessing

[Matt Newville]

On Thu, Dec 19, 2013 at 6:24 AM,  <josef.pktd at gmail.com> wrote:
>
>
>
> On Thu, Dec 19, 2013 at 4:55 AM, Frédéric Parrenin
> <parrenin at ujf-grenoble.fr> wrote:
>>
>> Dear all,
>>
>> Following these posts:
>>
>> http://stackoverflow.com/questions/10489134/multithreaded-calls-to-the-objective-function-of-scipy-optimize-leastsq
>> It seems it is possible to make leastsq take part of multiple processors.
>>
>> I was wondering: given that the tendency of processors is to have more and
>> more cores nowadays, why this is not done by default in leastsq?
>
>
> I think parallelizing leastsq would almost always be the wrong place to
> parallelize.

I am slightly reluctant to speak up, but I think this may not be true.

For calls to leastsq() with finite-difference Jacobians, MINPACK's
lmdif() calls fdjac2() in each iteration.  This subroutine then calls
the users objective function N times (for N variables) in a simple
loop with slightly different values for the variables.  Although these
calls share a work array this is an implementation detail and the
array elements per variable are actually independent.  This loop of N
calls to the objective function per iteration would be a good
candidate for a multiprocessing pool, and doing so could give a
substantial speed up for problems with more than a couple variables
and where the calculation of the objective function is the bottleneck
(which is typical for all but simple examples).

Currently, scipy's leastsq() simply calls the Fortran lmdif() (for
finite-diff Jacobian).   I think replacing fdjac2() with a
multiprocessing version would require reimplementing both lmdif() and
fdjac2(), probably using cython.  If calls to MINPACKs lmpar() and
qrfac() could be left untouched, this translation does not look too
insane -- the two routines lmdif() and fdjac2() themselves are not
that complicated.   It would be a fair amount of work, and I cannot
volunteer to do this myself any time soon.  But, I do think it
actually would improve the speed of leastsq() for many use cases.

Hoping this will inspire someone.....

--Matt