[SciPy-dev] the state of scipy unit tests

[Jarrod Millman]

On Mon, Nov 24, 2008 at 12:26 AM, Matthieu Brucher
<matthieu.brucher at gmail.com> wrote:
> There may be issues with this when people modify some packages deeply,
> but in a way not caught by the standard test battery. And then, when
> you go to alpha, you get hundreds of failing tests, and it's so
> overwhelming that you have to start the test battery from scratch.

I don't think that is very likely.  All the tests are still there and
would be run if you ran scipy.test('full').  I would imagine that many
people would run the full test-suite regularly.  A developer could run
scipy.test() for every change they make (if it only takes a short
amount of time) and then run scipy.test('full') just once or twice a
day.

> It could be better to have a scipy buildbot, like for numpy, that runs
> all the tests, and people before committing just check the most
> important tests. This way, you don't get hundreds of failing tests
> once you reactivate them, you can still track where the errors come
> from, and the test time for a single developer remains small
> (although, he could only check the result of the tests on the package
> he's modifying).

I think this should be done any way; but I don't think it solves the
problem for developers who want to quickly run the default test suite
regularly.  I think that decorating more of the tests as slow would
solve this problem.

The other problem is that we want binaries of alpha, beta, and rc
releases to run the full test suite by default, since, in this
instance, time isn't as important but completeness is.  This could be
solved by, for instance, by changing label to be full by default in
nosetester.py for tagged releases.  This could be done by running a
script that takes care of it or by adding some logic that changes the
behavior if a flag (e.g., release in version.py) is True in
version.py.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/