[SciPy-Dev] Global Optimization Benchmarks

[Ralf Gommers]

On Fri, Jan 8, 2021 at 11:35 AM Andrea Gavana <andrea.gavana at gmail.com>
wrote:

> Hi Ralf,
>
> On Fri, 8 Jan 2021 at 11:07, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>>
>>
>> On Fri, Jan 8, 2021 at 10:21 AM Andrea Gavana <andrea.gavana at gmail.com>
>> wrote:
>>
>>> Dear SciPy Developers & Users,
>>>
>>>     long time no see :-) . I thought to start 2021 with a bit of a bang,
>>> to try and forget how bad 2020 has been... So I am happy to present you
>>> with a revamped version of the Global Optimization Benchmarks from my
>>> previous exercise in 2013.
>>>
>>
>> Hi Andrea, this is awesome! Thanks for sharing!
>>
>
> I am happy you like it :-) .
>
>
>
>> This could be really useful to link to and use as a guide for providing
>> recommendations for solvers to use in the scipy.optimize tutorials. It's
>> good to see that SciPy overall is much more competitive than it was in
>> 2013. Overall it seems SHGO is our most accurate solver, and making it
>> faster seems worthwhile. That shouldn't be very difficult, given that it's
>> all pure Python still.
>>
>
> I have to say that, compared to back in 2013, the addition of SHGO and
> DualAnnealing to SciPy has made the global optimization world in SciPy much
> more powerful, pretty much at the top of what can currently be done with
> open source solvers.
>
>
>> MCS isn't open source, but both DIRECT and BiteOpt are MIT-licensed and
>> seem the best candidates to be considered for inclusion in SciPy.
>>
>
> I couldn't find a license restriction for MCS, but maybe I haven't looked
> hard enough... Do you have a link for it? I am just curious.
>

MCS itself doesn't contain any license information, but it depends on MINQ
which has a link in "All versions of MINQ are licensed" on this page:
https://www.mat.univie.ac.at/~neum/software/minq/. It's only free for
non-commercial use.


>
>> If you have recommendations or takeaways from all this work for SciPy,
>> I'd love to hear them.
>>
>
> I have a few thoughts, but please bear in mind that it's my opinion and
> only based on this exercise plus a couple of real-life problems I have been
> working on recently:
>

Thanks!


> 1. Assuming we are dealing with a low dimensional problem, SHGO is close
> to unbeatable. I have found a glitch when the number of variables gets 10+,
> or for repeated continuous optimizations, SHGO seems to require enormous
> amounts of memory: in the SciPy Extended benchmark, trying all the 100
> restarts got my RAM consumption to 190 GB for SHGO only - not something you
> want to ty unless you have a monster machine like mine.
>

That seems like something we should improve.

Cheers,
Ralf


> 2. DualAnnealing is also extremely powerful - ranking consistently close
> to the top for most of the benchmarks. It probably requires some tuning of
> the parameters (which I haven't done), especially when the allowable number
> of functions evaluations is large. That said, you can clearly see
> DualAnnealing shining in the SciPy Extended, GKLS, LGMVG and RandomFields
> benchmarks.
>
> 3. BasinHopping and DifferentialEvolution are generally slightly weaker,
> at least on these benchmarks. That said, I have used both of them with
> great success on real life problems - albeit with generally generous
> budgets of functions evaluations.
>
> 4. Real-life-wise, I recently had three very tough problems to work on:
> one 9-dimensional objective function describing multi-phase decline curves
> for oil/gas wells, which I am now satisfactorily fitting with
> DualAnnealing. Another one on optimization of 3D well trajectories, which I
> am happily handing over to SHGO or MCS depending on the problem. And
> another one related to wind and renewable data fitting which
> DifferentialEvolution is handling quite well.
>
> So, all in all, benchmarks only give you so much information: real-life
> problems sometimes defy the accepted wisdom that occurs because of
> contrived (but synthetic) objective functions. That said, if I had to
> attack a new problem and I had no idea where to start, I would definitely
> give SHGO and DualAnnealing the first go, as they are quite powerful across
> a large spectrum of problems.
>
> In the end, I believe that SciPy will definitely benefit from the addition
> of a couple (few?) more robust global solvers, especially if they
> implement techniques that are completely different from the existing ones
> (such as DIRECT, MCS, BiteOpt, of course). Giving more options to users is
> always going to make people happy - but of course you have to balance it
> with the maintenance efforts in the library.
>
> Andrea.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210108/38b50082/attachment-0001.html>