[SciPy-Dev] Global Optimization Benchmarks

[Andrea Gavana]

Hi Ralf,

On Fri, 8 Jan 2021 at 11:07, Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Fri, Jan 8, 2021 at 10:21 AM Andrea Gavana <andrea.gavana at gmail.com>
> wrote:
>
>> Dear SciPy Developers & Users,
>>
>>     long time no see :-) . I thought to start 2021 with a bit of a bang,
>> to try and forget how bad 2020 has been... So I am happy to present you
>> with a revamped version of the Global Optimization Benchmarks from my
>> previous exercise in 2013.
>>
>
> Hi Andrea, this is awesome! Thanks for sharing!
>

I am happy you like it :-) .



> This could be really useful to link to and use as a guide for providing
> recommendations for solvers to use in the scipy.optimize tutorials. It's
> good to see that SciPy overall is much more competitive than it was in
> 2013. Overall it seems SHGO is our most accurate solver, and making it
> faster seems worthwhile. That shouldn't be very difficult, given that it's
> all pure Python still.
>

I have to say that, compared to back in 2013, the addition of SHGO and
DualAnnealing to SciPy has made the global optimization world in SciPy much
more powerful, pretty much at the top of what can currently be done with
open source solvers.


> MCS isn't open source, but both DIRECT and BiteOpt are MIT-licensed and
> seem the best candidates to be considered for inclusion in SciPy.
>

I couldn't find a license restriction for MCS, but maybe I haven't looked
hard enough... Do you have a link for it? I am just curious.


> If you have recommendations or takeaways from all this work for SciPy, I'd
> love to hear them.
>

I have a few thoughts, but please bear in mind that it's my opinion and
only based on this exercise plus a couple of real-life problems I have been
working on recently:

1. Assuming we are dealing with a low dimensional problem, SHGO is close to
unbeatable. I have found a glitch when the number of variables gets 10+, or
for repeated continuous optimizations, SHGO seems to require enormous
amounts of memory: in the SciPy Extended benchmark, trying all the 100
restarts got my RAM consumption to 190 GB for SHGO only - not something you
want to ty unless you have a monster machine like mine.

2. DualAnnealing is also extremely powerful - ranking consistently close to
the top for most of the benchmarks. It probably requires some tuning of the
parameters (which I haven't done), especially when the allowable number of
functions evaluations is large. That said, you can clearly see
DualAnnealing shining in the SciPy Extended, GKLS, LGMVG and RandomFields
benchmarks.

3. BasinHopping and DifferentialEvolution are generally slightly weaker, at
least on these benchmarks. That said, I have used both of them with great
success on real life problems - albeit with generally generous budgets of
functions evaluations.

4. Real-life-wise, I recently had three very tough problems to work on: one
9-dimensional objective function describing multi-phase decline curves for
oil/gas wells, which I am now satisfactorily fitting with DualAnnealing.
Another one on optimization of 3D well trajectories, which I am happily
handing over to SHGO or MCS depending on the problem. And another one
related to wind and renewable data fitting which DifferentialEvolution is
handling quite well.

So, all in all, benchmarks only give you so much information: real-life
problems sometimes defy the accepted wisdom that occurs because of
contrived (but synthetic) objective functions. That said, if I had to
attack a new problem and I had no idea where to start, I would definitely
give SHGO and DualAnnealing the first go, as they are quite powerful across
a large spectrum of problems.

In the end, I believe that SciPy will definitely benefit from the addition
of a couple (few?) more robust global solvers, especially if they
implement techniques that are completely different from the existing ones
(such as DIRECT, MCS, BiteOpt, of course). Giving more options to users is
always going to make people happy - but of course you have to balance it
with the maintenance efforts in the library.

Andrea.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210108/05013ab5/attachment.html>