[SciPy-User] int32 overflow and sqrt of -ve number in scipy.stats.wilcoxon

[josef.pktd at gmail.com]

On Fri, Mar 22, 2013 at 8:43 AM, Simon Baldwin <simonb at google.com> wrote:
> I have run into two distinct but possibly connected failure modes in
> scipy.stats.wilcoxon.  Before bug-reporting, I'd appreciate any
> thoughts on these.
>
> The first is an int32 overflow that occurs where a count of data ties
> exceeds 46341.  This occurs in my real data set, comprising around
> 0.7M samples.  The cause is 32-bit counts returned by dfreps() in
> futilmodule.c.  Minimal failing demonstration:
>
> import numpy
> import scipy.stats
> numpy.seterr(all='raise')
> # Raises: FloatingPointError: overflow encountered in int_scalars,
> morestats.py, line 1242
> scipy.stats.wilcoxon([0.1] * 46341)  # 46341^2 > 2^31-1

this sounds like the same problem that rankdata had and Warren recently fixed.

My guess is that we can implement wilcoxon using the new rankdata
(maybe, if we can get the tie counts).
Otherwise, it would need a change to the c code.

(old code that wasn't designed for modern computers ?)


>
> The second arises from an attempt to take sqrt of a negative number.
> Again, this is in wilcoxon data tie handling and occurs in real data.
> Demonstration:
>
> import numpy
> import scipy.stats
> numpy.seterr(all='raise')
> # Raises: FloatingPointError: invalid value encountered in sqrt,
> morestats.py, line 1244
> scipy.stats.wilcoxon([0.1] * 10)

That might be a numerical precision issue in a corner case (-0) that should be
explicitly handled.

Please open a ticket.

Josef

>
> Both on scipy version 0.9, the latest I have access to.  I've taken a
> quick look at the 0.11 code for wilcoxon, and didn't notice any
> visible functional change.
>
> Thanks.
>
> --
> Google UK Limited | Registered Office: Belgrave House, 76 Buckingham
> Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user