[SciPy-Dev] Rationalisation of finite differences for minimize

[Matt Haberland]

1, 2, and 5-8 look good to me.

3) if `jac is None` or `bool(jac)` evaluates to `False`, then use forward
> differences with an absolute step of epsilon. This behaviour ensures back
> compatibility is kept for those methods that already accept an `epsilon` or
> `eps` keyword.
>

I would suggest that if `eps` is explicitly defined by the user, then the
value should be respected, but if it is not explicitly defined and
`bool(jac)` is `False`, we might upgrade the derivative approximation to
approx_derivative. If this is deemed a real backwards compatibility issue,
never mind.


> 4) have an additional `finite_diff_rel_step` keyword. For further
> information see `optimize._numdiff.approx_derivative`. This would be
> changed behaviour for some methods. trust-constr already has this keyword
> (which is why I suggested it). This keyword would only be added to those
> that already have an `epsilon` or `eps` keyword.
>

What will happen if the user specifies both? Rather than having a new
argument, I might have suggested that we use a provided `eps` as a relative
step size if `jac` is one of the allowed strings. Then again, if
`finite_diff_rel_step` is already used by `trust-constr`, it makes sense to
use it.

6) there are some methods that only work if points 1 or 2 are satisfied
> (i.e. `callable(jac) or jac is True`). These methods are: trust-krylov,
> trust-ncg, trust-exact, dogleg, Newton-CG (I hope I have them all). They
> raise an exception if a callable gradient function is not provided by the
> user.
> It's unclear to me if these methods can use a gradient calculated by
> finite differences. If they can, then I propose to add the ability to use
> finite differences with points 1, 2, 4, 5 (not 3) above. I need feedback on
> this point from subject experts.
>

If you don't hear back from the experts, I'd say it's worth a shot to try
adding this ability to use finite difference approximations to these
algorithms. Some algorithms are more robust to derivative approximation
errors than others, sure, but I've not heard of an algorithm that requires
its derivatives to be accurate to same machine precision as the function
evaluation.

Drawbacks
> ---------------
> g) Technically this meets back compatibility according to documentation.
> However, some users may be using jac=`2-point`, which at the moment may be
> converted to an absolute step+forward difference calculation. The
> documentation doesn't say that an absolute step is being used internally.
> The proposed behaviour would move it to a relative step. Thus users would
> get a slightly changed implementation. Backwards compatibility is only
> assured by setting `jac=False` or `jac=None`.
>
(If we think the suggestion for 3 above is OK, then backwards compatibility
is assured by setting  `jac=False` or `jac=None` AND explicitly defining
`eps`)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190821/fdd19a56/attachment.html>