[SciPy-Dev] CUTEst in Scipy

[Ralf Gommers]

On Tue, Mar 21, 2017 at 12:35 PM, Matt Newville <newville at cars.uchicago.edu>
wrote:

>
>
> Date: Tue, 21 Mar 2017 09:21:59 +1100
>> From: Andrew Nelson <andyfaff at gmail.com>
>> To: SciPy Developers List <scipy-dev at scipy.org>
>> Subject: Re: [SciPy-Dev] CUTEst in Scipy
>> Message-ID:
>>         <CAAbtOZc2JnOvgvzH4jEB3mpLBm_YZNCj9xy1q_PG2YN2+EMD4w at mail.gm
>> ail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> There are some least squares problems from the NIST test suite in the
>> benchmarks/benchmarks/go_benchmark_functions, but they're couched in
>> terms
>> of a simple scalar (chi2) minimisation problem, and they are all bounded
>> problems.
>>
>>
> The NIST StRD test suite (http://www.itl.nist.gov/
> div898/strd/nls/nls_main.shtml) could be considered for inclusion as a
> test suite for scipy.optimize.  These are generally small but non-trivial
> problems for unconstrained non-linear optimization. Each of the 20+ data
> sets comes with 2 sets of starting values for the variables and certified
> values for both best fit values and estimated uncertainties.  I believe
> that none of the optimizers in scipy.optimize will get the right answer to
> published precision in every case, though leastsq and least_squares can
> solve most of these problems reasonably well from at least 1 of the 2
> starting points.  This may imply that these problems are not "sparse" in
> the sense of what (as far as I understand) the GSOC project intends to
> focus on.  But, they are a good set of test cases to include in any
> assessment of an optimization algorithm.
>
> The NIST StRD datasets are public domain.  There is existing test code in
> lmfit to use these data sets with most of the scipy optimizers.  This could
> easily be modified to be part of scipy.   See https://github.com/lmfit/
> lmfit-py/tree/master/NIST_STRD and https://github.com/lmfit/
> lmfit-py/blob/master/tests/test_NIST_Strd.py for more details.  Also,
> note that the tests focus only on the quality of the results.  They may
> record number of iterations, but make no attempt to benchmark the runtime.
>

That may be a good thing - I think that there's in some places a lack of
verified accuracy tests while there are checks on no increases in number of
iterations. In scipy.stats we also have NIST tests for anova and
linregress, those were quite useful.


> I think that migrating this test suite to scipy would not be a huge
> undertaking, though I cannot say whether that is an appropriate use of time
> for the proposed GSOC project.
>

Can you open an issue for this to not lose track of the idea? Could be a
nice extra for GSoC or done separately.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170326/97d26ab1/attachment.html>