[SciPy-dev] Re: Fundamental scipy testing changes
Todd Miller
jmiller at stsci.edu
Tue Dec 7 18:11:36 EST 2004
The discussion below relates to porting scipy to numarray, starting with
some small changes to scipy_test.testing. The discussion began as
private e-mail but has crossed over to scipy-dev at Pearu's suggestion.
On Tue, 2004-12-07 at 16:16, Pearu Peterson wrote:
> Hi Todd,
>
> On 7 Dec 2004, Todd Miller wrote:
>
> > I took a closer look at how my scipy_base changes affect scipy.test()
> > and found a couple things in my scipy_base changes that needed fixing.
> > Sorry I didn't think of this sooner. These fixes are committed now on
> > the numarray branch in CVS. I also honed in on what I think should be
> > the starting point for discussion: the changes I made to
> > scipy_test/testing.py.
>
> Note that assert_equal, assert_almost_equal, assert_approx_equal
> were not meant to be used with array arguments (I didn't implement them
> but its obvious from reading the code).
Thanks for pointing this out... I noticed the array versions only
peripherally and didn't understand the distinction.
> For checking the equality of
> array arguments assert_array_equal or assert_array_almost_equal should be
> used. If some scipy test suite uses assert_equal, etc with array
> arguments then I think this is a bug of this particular test suite,
> not of testing.py. So, using scipy_base.all in assert_equal, etc is not
> necessary (unless we want to drop assert_array_* functions).
Understood. Are we agreed that it is appropriate to use all() in the
assert_array_* functions?
> > In order to do the numarray scipy_base changes and pass
> > scipy_base.test(), I modified scipy_test/testing.py in ways which I
> > assert are good. Mostly, I used all() in contexts where an array result
> > was being used as a truth value. In those contexts, the array
> > __nonzero__() function is executed. NumArray.__nonzero__() raises an
> > exception causing many tests to fail that should succeed. Using all()
> > reduces the "truth array" to a single scalar value, so
> > NumArray.__nonzero__() is never called.
> >
> > Where I think early agreement needs to happen is that my testing.py
> > changes are good even though they expose a handful of latent scipy bugs
> > or unit test bugs because Numeric's __nonzero__() has the meaning of
> > any() and not all(). So, for say an equality test, if any of the
> > values were equal, the test would succeed even if most of the arrays'
> > values were *not* equal. Switching to all() as I am advocating exposes
> > hidden problems like these.
>
> At the moment I have no comments on this as I haven't tried the numarray
> branch of scipy yet.
I think we should square away the scipy_test.testing changes before
anyone messes with the numarray branch.
> Btw, your discussion about the changes to scipy
> sounds reasonable. However, why do you say that using alltrue is an error?
> I thought all(x) is equivalent to alltrue(ravel(x)) and in assert_array_*
> functions the arguments to alltrue are already ravel'ed.
I looked at my diffs more carefully and see that you're right, the
alltrues are not bugs because they're only fed 1D arrays. I have not
reexamined all my scipy_base changes for instances of alltrue.
> > If my testing.py changes are agreed upon and the exposed bugs are either
> > fixed or acknowledged as known, we have a better basis for examining
> > the rest of the numarray changes to scipy_base.
> >
> > I attached 3 files for you to look at:
> >
> > 1. testing.diffs (changes to -r HEAD of scipy_test/testing.py)
> >
> > 2. results.HEAD (scipy.test() results against the HEAD of CVS using
> > Numeric)
> > I got 4 failures and can't see the nightly scoreboard yet so I don't
> > know if these are expected.
> >
> > 3. results.testing.changes (scipy.test() results with testing.diffs
> > applied)
> > I got 11 failures (including the original 4) most of which I believe are
> > either bugs or unit test bugs.
>
> The failures, where you see almost equal arrays, may occur when using
> Fortran blas;
My scipy.test() does complain about not finding clapack so maybe I am
using a Fortran blas too.
> these errors should be fixed by using proper decimal
> argument in assert_array_almost_equal call of the corresponding tests (I
> thought I have fixed these errors in scipy HEAD already).
As you said I think, the (incorrect) changes to assert_equal exposed
(incorrect) uses of assert_equal in array contexts.
> Other failures need more attention.
>
> > If any of you would rather not be included in my future numarray
> > mailings, or you have suggestions for a better venue for them, please
> > let me know.
>
> I suggest using scipy-dev.
>
> Thanks,
> Pearu
I re-attached the attachments in case anyone on scipy-dev wants to look
as well; there's nothing new there.
Thanks for looking it over,
Todd
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testing.diffs
Type: text/x-patch
Size: 5846 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20041207/a3842097/attachment.bin>
-------------- next part --------------
======================================================================
FAIL: check_heevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 80, in check_heevr_irange_high
def check_heevr_irange_high(self): self.check_syevr_irange(sym='he',irange=[1,2])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269+0.j 4.0951085+0.j 7.3420534+0.j]
Array 2: [ 3.6929276+0.j 4.0951101+0.j 7.3420528+0.j]
======================================================================
FAIL: check_heevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 102, in check_heevr_vrange
def check_heevr_vrange(self): self.check_syevr_vrange(sym='he')
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269+0.j 4.0951085+0.j 7.3420534+0.j]
Array 2: [ 3.6929276+0.j 4.0951101+0.j 7.3420528+0.j]
======================================================================
FAIL: check_syevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 74, in check_syevr_irange_high
def check_syevr_irange_high(self): self.check_syevr_irange(irange=[1,2])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269 4.0951085 7.3420534]
Array 2: [ 3.6929276 4.0951101 7.3420528]
======================================================================
FAIL: check_syevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 742, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269 4.0951085 7.3420534]
Array 2: [ 3.6929276 4.0951101 7.3420528]
----------------------------------------------------------------------
Ran 1172 tests in 7.154s
FAILED (failures=4)
<unittest.TextTestRunner object at 0x4189b3b4>
[413475 refs]
-------------- next part --------------
======================================================================
FAIL: check_basic (scipy_base.function_base.test_function_base.test_amax)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_base/tests/test_function_base.py", line 72, in check_basic
assert_equal(amax(b),[8.0,10.0,9.0])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: [8.0, 10.0, 9.0]
ACTUAL: array([ 9., 10., 8.])
======================================================================
FAIL: check_basic (scipy_base.function_base.test_function_base.test_amin)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_base/tests/test_function_base.py", line 82, in check_basic
assert_equal(amin(b),[3.0,3.0,2.0])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: [3.0, 3.0, 2.0]
ACTUAL: array([ 3., 4., 2.])
======================================================================
FAIL: check_heevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 80, in check_heevr_irange_high
def check_heevr_irange_high(self): self.check_syevr_irange(sym='he',irange=[1,2])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269+0.j 4.0951085+0.j 7.3420534+0.j]
Array 2: [ 3.6929276+0.j 4.0951101+0.j 7.3420528+0.j]
======================================================================
FAIL: check_heevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_complex)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 102, in check_heevr_vrange
def check_heevr_vrange(self): self.check_syevr_vrange(sym='he')
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269+0.j 4.0951085+0.j 7.3420534+0.j]
Array 2: [ 3.6929276+0.j 4.0951101+0.j 7.3420528+0.j]
======================================================================
FAIL: check_syevr_irange_high (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 74, in check_syevr_irange_high
def check_syevr_irange_high(self): self.check_syevr_irange(irange=[1,2])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 68, in check_syevr_irange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269 4.0951085 7.3420534]
Array 2: [ 3.6929276 4.0951101 7.3420528]
======================================================================
FAIL: check_syevr_vrange (scipy.lib.lapack.test_lapack.test_flapack_float)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/lib/lapack/tests/esv_tests.py", line 94, in check_syevr_vrange
assert_array_almost_equal(dot(a,v[:,i]),w[i]*v[:,i])
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 744, in assert_array_almost_equal
assert cond,\
AssertionError:
Arrays are not almost equal (mismatch 33.3333333333%):
Array 1: [ 3.6929269 4.0951085 7.3420534]
Array 2: [ 3.6929276 4.0951101 7.3420528]
======================================================================
FAIL: check_arange (scipy.special.basic.test_basic.test_arange)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/special/tests/test_basic.py", line 495, in check_arange
assert_equal(numstring,array([0.,0.1,0.2,0.3,
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
======================================================================
FAIL: check_genlaguerre (scipy.special.basic.test_basic.test_laguerre)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/special/tests/test_basic.py", line 1573, in check_genlaguerre
assert_equal(lag2.c,array([1,-2*(k+2),(k+1.)*(k+2.)])/2.0)
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: array([ 0.5 , -3.58365527, 4.62946491])
ACTUAL: array([ 0.5 , -3.58365527, 4.62946491])
======================================================================
FAIL: check_legendre (scipy.special.basic.test_basic.test_legendre)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/special/tests/test_basic.py", line 1590, in check_legendre
assert_equal(leg3.c,array([5,0,-3,0])/2.0)
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: array([ 2.5, 0. , -1.5, 0. ])
ACTUAL: array([ 2.50000000e+00, 0.00000000e+00, -1.50000000e+00, 3.10191936e-17])
======================================================================
FAIL: check_diag (scipy.linalg.basic.test_basic.test_tri)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/linalg/tests/test_basic.py", line 427, in check_diag
assert_equal(tri(4,k=1),array([[1,1,0,0],
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
DESIRED: array([[1, 1, 0, 0],
[1, 1, 1, 0],
[0, 1, 1, 1],
[1, 1, 1, 1]])
ACTUAL: array([[1, 1, 0, 0],
[1, 1, 1, 0],
[1, 1, 1, 1],
[1, 1, 1, 1]],'b')
======================================================================
FAIL: check_diag (scipy.linalg.basic.test_basic.test_triu)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jmiller/work/lib/python2.4/site-packages/scipy/linalg/tests/test_basic.py", line 494, in check_diag
assert_equal(tril(a,k=-2),b)
File "/home/jmiller/work/lib/python2.4/site-packages/scipy_test/testing.py", line 649, in assert_equal
assert _sb.all(desired == actual), msg
AssertionError:
Items are not equal:
----------------------------------------------------------------------
Ran 1172 tests in 6.999s
FAILED (failures=11)
<unittest.TextTestRunner object at 0x4189c424>
[455000 refs]
>>>
More information about the SciPy-Dev
mailing list