[SciPy-Dev] CUTEst in Scipy

[Andrew Nelson]

Many of the NIST tests are in the global optimisation benchmark functions.
It might be a good idea to have them all in one place though.

On 26 Mar 2017 1:14 pm, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:

>
>
> On Tue, Mar 21, 2017 at 12:35 PM, Matt Newville <
> newville at cars.uchicago.edu> wrote:
>
>>
>>
>> Date: Tue, 21 Mar 2017 09:21:59 +1100
>>> From: Andrew Nelson <andyfaff at gmail.com>
>>> To: SciPy Developers List <scipy-dev at scipy.org>
>>> Subject: Re: [SciPy-Dev] CUTEst in Scipy
>>> Message-ID:
>>>         <CAAbtOZc2JnOvgvzH4jEB3mpLBm_YZNCj9xy1q_PG2YN2+EMD4w at mail.gm
>>> ail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> There are some least squares problems from the NIST test suite in the
>>> benchmarks/benchmarks/go_benchmark_functions, but they're couched in
>>> terms
>>> of a simple scalar (chi2) minimisation problem, and they are all bounded
>>> problems.
>>>
>>>
>> The NIST StRD test suite (http://www.itl.nist.gov/div89
>> 8/strd/nls/nls_main.shtml) could be considered for inclusion as a test
>> suite for scipy.optimize.  These are generally small but non-trivial
>> problems for unconstrained non-linear optimization. Each of the 20+ data
>> sets comes with 2 sets of starting values for the variables and certified
>> values for both best fit values and estimated uncertainties.  I believe
>> that none of the optimizers in scipy.optimize will get the right answer to
>> published precision in every case, though leastsq and least_squares can
>> solve most of these problems reasonably well from at least 1 of the 2
>> starting points.  This may imply that these problems are not "sparse" in
>> the sense of what (as far as I understand) the GSOC project intends to
>> focus on.  But, they are a good set of test cases to include in any
>> assessment of an optimization algorithm.
>>
>> The NIST StRD datasets are public domain.  There is existing test code in
>> lmfit to use these data sets with most of the scipy optimizers.  This could
>> easily be modified to be part of scipy.   See
>> https://github.com/lmfit/lmfit-py/tree/master/NIST_STRD and
>> https://github.com/lmfit/lmfit-py/blob/master/tests/test_NIST_Strd.py
>> for more details.  Also, note that the tests focus only on the quality of
>> the results.  They may record number of iterations, but make no attempt to
>> benchmark the runtime.
>>
>
> That may be a good thing - I think that there's in some places a lack of
> verified accuracy tests while there are checks on no increases in number of
> iterations. In scipy.stats we also have NIST tests for anova and
> linregress, those were quite useful.
>
>
>> I think that migrating this test suite to scipy would not be a huge
>> undertaking, though I cannot say whether that is an appropriate use of time
>> for the proposed GSOC project.
>>
>
> Can you open an issue for this to not lose track of the idea? Could be a
> nice extra for GSoC or done separately.
>
> Ralf
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170326/9fb1ea77/attachment.html>