[SciPy-Dev] GSoC'21 participation SciPy

[Andrew Nelson]

It's good to hear about the ask-tell interface, it's not something I'd
heard about before.

The class-based Optimizer that was proposed wasn't going to work in quite
that way. The main concept was to create an (e.g.) LBFGSB class (inheriting
a Minimizer superclass). All Minimizer objects would be iterators, having a
__next__ method that would perform one step of a minimisation loop.
Iterator based design syncs quite well with the loop based design of most
of the existing minimisation algorithms. The __next__ method would be
responsible for calling the user based functions. If the user based
functions could be marked as vectorisable the __next__ method could
despatch a whole series of `x` locations for the user function (one or all
of func/jac/hess) to evaluate; the user function could do whatever
parallelisation it wanted. Vectorisable function evaluations also offer
benefits for numerical differentiation evaluation.
The return value of __next__ would be something along the lines of an
intermediate OptimizeResult.

I don't know the ask-tell approach works in finer detail. For example, each
minimisation step typically requires multiple function evaluations to
proceed, e.g. at least once for func evaluation, and many times more for
grad/jac and hess evaluation (not to mention constraint function
evaluations). THerefore there wouldn't be a 1:1 correspondence of a single
ask-tell and a complete step of the minimizer.

I reckon the development of this would be way more than a single GSOC could
provide, at least to get a mature design into scipy. It's vital to get the
architecture correct (esp. the base class), when considering all the
minimizers that scipy offers, and their different vagaries. Implementing
for one or two minimizers wouldn't be sufficient otherwise one forgets that
they e.g. all have different approaches to halting, and you find yourself
bolting other things on to make things work. In addition, it's not just the
minimizer methods that are involved, it's considering how this all ties in
with how constraints/numerical differentiation/`LowLevelCallable`/etc could
be improved/used in such a design. At least for the methods involved in
`minimize` such an opportunity is the time to consider a total redesign of
how things work. Smart/vectorisable numerical differentiation would be more
than a whole GSOC in itself.

As Robert says, implementation in a separate package would probably be the
best way to work; once the bugs have been ironed out it could be merged
into scipy-proper. Any redesign could take into account the existing
API's/functionality to make things a less jarring change.

It'd be great to get the original class-based Optimization off the ground,
or something similar. However, it's worth noting that the original proposal
only received lukewarm support.

A.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210218/1cc56127/attachment-0001.html>