[SciPy-Dev] Return residual and derivative in gradient descent methods like Newton

[Mark Alexander Mikofski]

Hi Josef,

Sorry, I didn't phrase my question very well.

I meant from the callback function, not the output from the solver.

For example, say we want to solve:

y = exp(Ax+y)

dy/dx = A*y

My proposal is when supplying arguments to the solver instead of writing
two callbacks we just supply one:

>>> def f(x, y, A):
...     # callback returns both fval and jac
...     F = y - exp(Ax+y)  # residual
...     J = -A*y  # jacobian
...     return F, J

>>> newton(f, x0, fprime='anystring', args=(y0, A))

'anything' indicated to the solver that `fprime` is returned by the
callback `f`. So inside the solver instead of calling them separately it
just makes one call:

...     try:
...         J = fprime(x, *args)
...     except Exception:
...         F, J = f(x, *args)
...     else:
...         F = f(x, *args)

This is a common API in MATLAB, and my tests show that for some functions
with this can reduce the number of calls and improve the speed of the
solver.

Maybe I could get the same improvements by precalculating some things and
passing them in `args`?

On Mon, Jun 25, 2018, 7:58 AM <josef.pktd at gmail.com> wrote:

> On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski
> <mikofski at berkeley.edu> wrote:
> > Hi,
> >
> > Would anyone disagree or would anyone be interested in a proposal to
> allow
> > the derivative to be returned from the user supplied function as an
> optional
> > second argument in gradient search method like Newton? EG
> >
> >>>> lambda x,a: (x**3-a, 3*x*"2)
> >>>> newton(f, x0, fprime='f2', args=(a,))
> >
> > Some simple tests show that this may have a 2X speed in cases where the
> > derivative expression requires the value of the original function call.
> >
> > See this issue in which I proposed this idea and wore a sample test
> script
> > https://github.com/scipy/scipy/issues/8354
> >
> > I proposed a way to keep the existing API and add this new feature.
> > * If `fprime` is a callable, then same as before
> > * If it's a string like "f2" the get fprime from second output
>
> I think using something like a `full_output` option as in the fmin_xxx
> function would be more explicit and more flexible.
> e.g. the second optional return could be a dict or similar.
>
> Josef
>
>
> >
> > Thanks,
> > Mark
> >
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
> >
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180625/2b0f4f46/attachment.html>