[SciPy-dev] Problem with F distribution, or with me? - error in stats.fatiguelife.rvs

[Anne Archibald]

2008/8/24  <josef.pktd at gmail.com>:

> Running the scipy test suite looks pretty useless for verifying the
> actual distribution except for serious mistakes. It didn't detect
> before anything wrong with fatiguelife or loggamma (which I think also
> gives incorrect random numbers)
>
> The Kolmogorov test in
> http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/stats/tests/test_distributions.py
> is pretty powerless to detect mistakes in the actual distribution.
> N=30 is too small and the fail threshold for the pval for fatiguelife
> is set to alpha = 0.01, while for the other distributions it is at
> alpha = 0.1.
>
> The pvalue for N=100 or N=1000  should be a much better indicator
> whether the random variable really follows the theoretical
> distribution.

Is there any reason not to include your fuzz tests as a part of scipy?
If they take too long, it may make sense to run them only when
exhaustive testing is requested, but they find a lot of bugs for such
a little bit of code.

Also, for the many tests of scipy that use random numbers, does it
make sense to seed the random number generator ahead of time? That way
debuggers can replicate the test failures... Perhaps nose provides a
way to seed the random number generator before every test?

Anne