[SciPy-User] leastsq and multiprocessing

[Frédéric Parrenin]

Dear all,

Coming back to an old thread...

I tried Jeremy's method since it is the easiest to implement.
Below is the Dfun function I provided to leastsq.
In my experiment, I used a pool of 6 since I have 8 cores in my PC.

However, the computer becomes extremely slow, almost unusable, during the
experiment.
Do you know why this happens?

Best regards,

Frédéric

________________

def Dres(var):
    """Calculate derivatives for each parameter using pool."""
    zeropred = residuals(var)
    derivparams = []
    results=[]
    delta = m.sqrt(np.finfo(float).eps) #Stolen from the leastsq code
    for i in range(len(var)):
        copy = np.array(var)
        copy[i] += delta
        derivparams.append(copy)
#        results.append(residuals(derivparams))
    if __name__ == "__main__":
        pool = multiprocessing.Pool(nb_nodes)
    results = pool.map(residuals, derivparams)
    derivs = [ (r - zeropred)/delta for r in results ]
    return derivs



2013-12-20 13:43 GMT+01:00 Jeremy Sanders <jeremy at jeremysanders.net>:

> Matt Newville wrote:
>
> > Currently, scipy's leastsq() simply calls the Fortran lmdif() (for
> > finite-diff Jacobian).   I think replacing fdjac2() with a
> > multiprocessing version would require reimplementing both lmdif() and
> > fdjac2(), probably using cython.  If calls to MINPACKs lmpar() and
> > qrfac() could be left untouched, this translation does not look too
> > insane -- the two routines lmdif() and fdjac2() themselves are not
> > that complicated.   It would be a fair amount of work, and I cannot
> > volunteer to do this myself any time soon.  But, I do think it
> > actually would improve the speed of leastsq() for many use cases.
>
> Computing the Jacobian using using multiprocessing definitely helps the
> speed. I wrote the unrated answer (xioxox) there which shows how to do it
> in
> Python.
>
> Jeremy
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140522/f94c94b6/attachment.html>