[SciPy-dev] Logical operations, Numeric.sum() and overflow

[Ed Schofield]

Hi all,

Importing 'scipy' changes the output of the following code:

>>>import random
>>>import Numeric, RandomArray
>>>(m,n) = (1000,3)
>>>x = RandomArray.random((m,n))
>>>y = x < 0.5
>>>assert sum(y) == Numeric.sum(y)

from nothing to an AssertionError.

This is random behaviour: the error occurs about 90% of the time with
this value of m on my PC (NumPy 23.1, SciPy 0.3.2, Python 2.4.1, Linux
2.6.11) , but reducing m to 500 makes it occur only about 20% of the time.

There appear to be two causes:
(1) importing scipy changes the behaviour of "y = x < 0.5" to return an
array of typecode 'b' rather than 'l'.
(2) Numeric.sum() is prone to overflow errors, returning an array of
type 'b' rather than increasing precision:

>>> a = Numeric.array([[253,254,255],[1,1,1]],'b')
>>> Numeric.sum(a)
array([254, 255,   0],'b')

Here's my two cents.  On point (1), unit tests are needed to ensure a
simple 'import scipy' can't change the behaviour of unrelated Numpy
code.  (How is this even possible?)

Point (2) seems to indicate a design flaw with Numeric.  How do Octave
and Matlab deal with this?  Whatever they do, it "just works", whereas
Numeric feels "broken" in this respect; this overflow propagates through
other operations (Numeric.average() in my case), and finding such bugs
can take hours.  Any suggestions / ideas?


-- Ed