[SciPy-dev] Problem with F distribution, or with me? - error in stats.fatiguelife.rvs

[josef.pktd at gmail.com]

On Tue, Aug 26, 2008 at 4:37 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:

>
> Is there any reason not to include your fuzz tests as a part of scipy?
> If they take too long, it may make sense to run them only when
> exhaustive testing is requested, but they find a lot of bugs for such
> a little bit of code.
>
> Also, for the many tests of scipy that use random numbers, does it
> make sense to seed the random number generator ahead of time? That way
> debuggers can replicate the test failures... Perhaps nose provides a
> way to seed the random number generator before every test?
>
> Anne
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>

I see two problems for including these tests when they are intended to
be powerful.

The main problem are false positives, i.e. test failures because the
random sample just gives by chance a failing test statistic.

If you seed all random variables, it takes the fuzziness out of these
tests. Currently the parameters of the distribution are randomly drawn
and then the kstests draw, I guess, a random sample for the actual
test. Only seeding one of them would not remove the possibility of
accidental test failures.

My tests are just a variation of the current test in scipy.stats, I
just included more distributions and increased the power of the tests,
which is now also changed in trunk. The pvalue thresholds are set
pretty low and when a test fails, then a second test is run, I assume
in order to reduce the number of false failures. If the tests are too
weak, then they won't detect anything except very severe errors. But
when running the test suite you don't want to get lots of spurious
test failures.

What I did with these test was, make them very strict and then check
individually whether a test failure is actually caused by an error or
just a weakness of the kstest. For some kstest with low pvalues, I
later did not find anything (obviously) wrong with the distribution in
scipy.stats.

For now, my strict fuzz tests are to coarse, all I know in general is
that some tests don't work for some parameters that are not ruled out.
Maybe these parameters should be ruled out for the current
implementation.

But my tests indicate that there might still be other problems for
scipy.stats to run according to it's own specification, and there are
no tests for many other methods or properties e.g. pdf or pmf,
statistics such as moments, at least not from what I have seen.

Can somebody run:

stats.zipf.pmf(range(10),1.5)
stats.zipf.cdf(10,1.5)
stats.zipf.cdf(range(10),1.5)

I get
>>> stats.zipf.cdf(10,1.5)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Josef\_progs\Subversion\scipy_trunk\dist\Programs\Python24\Lib\site-p
ackages\scipy\stats\distributions.py", line 3549, in cdf
    place(output,cond,self._cdf(*goodargs))
  File "C:\Josef\_progs\Subversion\scipy_trunk\dist\Programs\Python24\Lib\site-p
ackages\scipy\stats\distributions.py", line 3458, in _cdf
    return self._cdfvec(k,*args)
  File "C:\Programs\Python24\Lib\site-packages\numpy\lib\function_base.py", line
 1092, in __call__
    raise ValueError, "mismatch between python function inputs"\
ValueError: mismatch between python function inputs and received arguments

but this might be, because I am running either an older version of
scipy or from my own faulty (?) build from subversion.
If the current version still has this error, then I think it is
related to the generic calculation of the cdf,
similar to ticket 422, changeset 3797 for the moment calculation.

In my tests I get many value errors, but again I don't know whether it
is my version/setup, whether the parameters are not ruled out but
don't make sense, or whether there is actually a bug somewhere.

Josef