[SciPy-Dev] User Acceptance Testing

[josef.pktd at gmail.com]

On Sat, May 14, 2011 at 5:05 AM, Pauli Virtanen <pav at iki.fi> wrote:
> On Fri, 13 May 2011 19:40:44 -0400, Joshua Grant wrote:
>> Hi everyone. So I notice that for Scipy, there's some fairly extensive
>> unit tests, which are great for the actual development of the library
>> code. Are there any forms of user acceptance testing, automated or
>> otherwise? Ways to simulate user behaviour and larger blocks of code,
>> beyond single methods/functions?
>
> You can write a given functional test in the form of a unit test --
> just solve a problem as you would do normally, and put asserts at
> the end to check that the result is correct. In fact, this is what
> a majority of the unit tests in Scipy do; they're (fixed) functional
> tests for high-level routines.
>
> I'm not sure how you would generate tests automatically for library
> code (do you have some ideas :)? With UIs and web pages I can understand
> that one can do fuzz testing to simulate user input, and check that
> whatever is done does not cause totally unexpected results.  But when
> the user input is scientific code, the specifications are so complex that
> I don't see a way to generate that automatically --- it's easier to write
> out given cases manually.

Since scipy is a library of individual functions and classes, the unit
tests are functional tests, and acceptance tests, since that's what
the user is using, e.g. a function in scipy.special or stats or linalg
either produces a correct result or not. scipy.optimize manages to
optimize or not.

But it's all for some predesigned cases and users can come with
unusual edge cases, which are slow to catch. For scipy.stats I started
out with fuzz tests, throwing many different random parameters at the
distributions, but doing this on a regular basis as part of the test
suite takes too much testing time.

However, numpy and scipy are also tested against "larger blocks of
code, beyond single methods/functions" but not as part of the scipy
test suite.  In the release preparation everyone is encouraged to test
against there own code.

And for example tests of (at least) scikits.statsmodels and matplotlib
against the new numpy 1.6 found some bugs. The tests of bottleneck
found some strange or inconsistent edge cases. numpy has scipy as a
big user to be tested against.

But for example in the case of one bug in scikits.statsmodels with
incorrect casting, it took Skipper a while to figure out why the
precision of some test results dropped in one case. The problem is
better handled with a unit test, then trying to figure out why some
results after lots of different calculations have only 2 significant
digits precision instead of 4 decimal precision (or something like
this).

So, I think functional unit tests are fine, but coverage for some
parts of scipy is not great.
In some cases of failures in scipy.optimize, I'm not sure whether it's
bugs or inherent to the optimization problems. And as an acceptance
test, I have never heard of anyone running numpy.random through the
randomness test suites http://en.wikipedia.org/wiki/Diehard_tests .


Josef

>
> --
> Pauli Virtanen
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>