[SciPy-Dev] minimizer benchmark

[alex]

On Mon, Nov 18, 2013 at 10:20 AM, Jacob Stevenson
<jstevenson131 at gmail.com> wrote:
> Hi Everyone, it seemed to me that the folder scipy/optimize/benchmarks/
> was rather lonely, with only one file in it, so I wrote a script which
> benchmarks the scipy minimizers.  The script simpy runs all the
> optimizers on the Rosenbrock function and prints various information.
> Here's the output sorted by minimization time
>
> Optimizer benchmark on the Rosenbrock function sorted by time
>         L-BFGS-B pass nfev    6 njev    0 nhev    0 time 0.000691891
>              TNC pass nfev   13 njev    0 nhev    0 time 0.00105786
>        Newton-CG pass nfev   17 njev   27 nhev   11 time 0.00400996
>            SLSQP pass nfev   41 njev   27 nhev    0 time 0.00410509
>        trust-ncg pass nfev   18 njev   16 nhev   15 time 0.00415802
>           dogleg pass nfev   16 njev   14 nhev   13 time 0.00426602
>               CG pass nfev   63 njev   63 nhev    0 time 0.0065279
>             BFGS pass nfev   44 njev   44 nhev    0 time 0.0070231
>           Powell pass nfev  524 njev    0 nhev    0 time 0.0262001
>           COBYLA fail nfev 1000 njev    0 nhev    0 time 0.026603
>
> The results are interesting with L-BFGS-B outperforming all the others
> by a significant margin in both total time and total number of function
> calls.
>
> I have not submitted a pull request because I have no idea how to fit
> what I've done into an existing benchmarking framework (if there is
> one).  I will submit the pull request if there is interest.  In my
> opinion it would be really useful to have a long set of benchmarks to
> see how all the minimizers perform on different types of minimization
> problems.
>
> Here is the script in my scipy fork
>
> https://github.com/js850/scipy/blob/benchmarks/scipy/optimize/benchmarks/bench_optimizers.py
>
> Best wishes,
> Jake

I'll make a couple of comments, before I forget everything I learned
about Rosenbrock and about the scipy optimization framework when I
implemented the dogleg and trust-ncg functions from the Nocedal and
Wright book...

First, thanks for adding these benchmarks!  This seems like a natural
comparison to have available in scipy, and I suspect that this has not
been done earlier because Denis has only recently created a general
framework that allows interchangeable minimization functions.  I hope
that something like this will go into scipy.

I've written similar one-off benchmarking code for algopy
https://github.com/b45ch1/algopy/tree/master/documentation/sphinx/examples/minimization
before I'd started contributing to scipy.  This includes the
Rosenbrock function as well as a couple of other good benchmarking
functions, and easy and hard starting guesses for each; the licenses
are compatible so maybe this would be useful for future scipy
minimize() benchmarks.

Benchmarks are an endless source of nitpicking, so I'll mention two
nitpicks for your code :)  First, the different minimize() functions
might differ in how close they get to the optimum, before they report
that they have successfully converged.  The nth dimensional rosenbrock
function has its minimum at (1, 1, ..., 1) so this should be not so
hard to compare closeness.  Second and possibly more importantly, the
relative performances may change depending on whether the starting
guess is easy vs. hard.  Your starting point x0=[0.8, 1.2, 0.7] is
relatively near (1, 1, 1) so it is "easy".  Starting points that
require walking around the Rosenbrock valley
http://en.wikipedia.org/wiki/File:Rosenbrock_function.svg are more
tricky, and the less clever minimization functions may outperform the
more clever functions at easy starting guesses.

Best,
Alex