[SciPy-Dev] memory corruption when running scipy 0.8 test suite

Wed Jul 14 05:25:36 EDT 2010

On 7/13/2010 8:19 PM, Charles R Harris wrote:
>
>
> On Tue, Jul 13, 2010 at 4:52 PM, Christoph Gohlke <cgohlke at uci.edu
> <mailto:cgohlke at uci.edu>> wrote:
>
>     Hi Ralph,
>
>     On 7/13/2010 5:26 AM, Ralf Gommers wrote:
>     >  Hi Christoph,
>     >
>     >  On Tue, Jul 13, 2010 at 9:11 AM, Christoph Gohlke <cgohlke at uci.edu
>     <mailto:cgohlke at uci.edu>
>     >  <mailto:cgohlke at uci.edu <mailto:cgohlke at uci.edu>>> wrote:
>     >
>     >      Dear SciPy developers,
>     >
>     >      I am trying to fix the errors and failures reported in the thread
>     >  "[SciPy-User] many test failures on windows 64"
>     >
>     <http://mail.scipy.org/pipermail/scipy-user/2010-July/025961.html>. Most
>     >      of the issues seem specific for the msvc9/MKL build of scipy
>     0.8. I am
>     >      now down to 1 error and 6 failures (from 16 errors and 9
>     failures).
>     >
>     >
>     >  Thanks for working on this! Can you please share the fixes you have
>     >  already (git branch or patches)? I have to make another release
>     >  candidate which fixes the Rbf issue Scott Sinclair noticed and would
>     >  like to incorporate these changes.
>
>
>     I have opened tickets and left comments on NumPy and SciPy trac:
>
>     http://projects.scipy.org/scipy/ticket/1229
>     "Patch for two test failures with msvc9 build"
>
>
>     http://projects.scipy.org/scipy/ticket/1225
>     "Test errors in scipy.sparse when using MSVC/MKL build"
>     The sparse.linalg.spsolve function works correctly when linking SuperLU
>     against CBLAS instead of MKL. Nevertheless, one error and two failures
>     remain for the scipy.sparse tests.
>
>
>     http://projects.scipy.org/numpy/ticket/1539
>     "MSVC specific TypeError when using double, longdouble in numpy.dot"
>     This is fixed in numpy trunk.
>
>
>     http://projects.scipy.org/scipy/ticket/678
>     "scipy.test failure with mkl/cdft"
>     The scipy.odr test failures also appear on linux platforms when scipy is
>     linked with MKL.
>
>
>     http://projects.scipy.org/scipy/ticket/1210
>     "crash during sparse matrix slicing with Python 2.7"
>     This crash still persists with Python 2.7 final.
>
>
>     >
>     >  For the ndimage issue I'm not sure what the answer is, but
>     skipping the
>     >  dsyevr tests on 64-bit Windows only should at least allow you to build
>     >  binaries that pass all tests, right?
>
>     All ndimage tests would pass. One error and five failures would still
>     remain for the whole scipy test suite.
>
>     I am not suggesting to disable the dsyevr test in the final release.
>     Hiding the failure could create false confidence.
>
>
>
>     >      The following failure in ndimage does not appear when
>     ndimage.test() is
>     >      run out of context of scipy.test()
>     >
>     >      FAIL: extrema 3
>     >
>     ----------------------------------------------------------------------
>     >      Traceback (most recent call last):
>     >         File
>     >  "C:\Python26\lib\site-packages\scipy\ndimage\tests\test_ndimage.py",
>     >      line 3149, in test_extrema03
>     >           self.failUnless(numpy.all(output1[2]  == output4))
>     >      AssertionError
>     >
>     >
>     >      The output1[2] array contains a NaN in the first position. If
>     I disable
>     >      the following dsyevr related tests in
>     >  scipy.lib.lapack.tests.test_esv.py
>     <http://scipy.lib.lapack.tests.test_esv.py>
>     >  <http://scipy.lib.lapack.tests.test_esv.py>, all ndimage tests pass.
>     >      Could this
>     >      be a memory corruption issue in MKL? Besides the ndimage failure,
>     >      scipy.lib.lapack seems to work and passes all tests. Also,
>     this artifact
>     >      only happens or surfaces on the 64-bit build.
>     >
>     >
>     >      Index: test_esv.py
>     >
>     ===================================================================
>     >      --- test_esv.py (revision 6598)
>     >      +++ test_esv.py (working copy)
>     >      @@ -91,17 +91,17 @@
>     >            def test_ssyevr(self):
>     >                self._test_base('ssyevr', 'F')
>     >
>     >      -    @dec.skipif(FLAPACK_IS_EMPTY, "Flapack empty, skip
>     flapack test")
>     >      -    def test_dsyevr(self):
>     >      -        self._test_base('dsyevr', 'F')
>     >      +#    @dec.skipif(FLAPACK_IS_EMPTY, "Flapack empty, skip
>     flapack test")
>     >      +#    def test_dsyevr(self):
>     >      +#        self._test_base('dsyevr', 'F')
>     >
>     >            @dec.skipif(FLAPACK_IS_EMPTY, "Flapack empty, skip
>     flapack test")
>     >            def test_ssyevr_ranges(self):
>     >                self._test_syevr_ranges('ssyevr', 'F')
>     >
>     >      -    @dec.skipif(FLAPACK_IS_EMPTY, "Flapack empty, skip
>     flapack test")
>     >      -    def test_dsyevr_ranges(self):
>     >      -        self._test_syevr_ranges('dsyevr', 'F')
>     >      +#    @dec.skipif(FLAPACK_IS_EMPTY, "Flapack empty, skip
>     flapack test")
>     >      +#    def test_dsyevr_ranges(self):
>     >      +#        self._test_syevr_ranges('dsyevr', 'F')
>     >
>     >            # Clapack tests
>     >            @dec.skipif(CLAPACK_IS_EMPTY or not FUNCS_CLAPACK["ssyev"],
>     >
>     >
>     >      I checked the flapack_esv.pyf.src code but could not find anything
>     >      obvious. I am linking against mkl_lapack95_lp64, mkl_blas95_lp64,
>     >      mkl_intel_lp64, mkl_intel_thread, and mkl_core, MKL version
>     10.2.5.1.
>     >
>
>
>
>     This is another weird test failure that is apparently dependent on the
>     context in which the test is run:
>
>     ======================================================================
>     FAIL: Real-valued Bessel domains
>     ----------------------------------------------------------------------
>     Traceback (most recent call last):
>        File
>     "X:\Python26\lib\site-packages\scipy\special\tests\test_basic.py", line
>     1691, in test_ticket_854
>          assert not isnan(airye(-1)[2:4]).any(), airye(-1)
>     AssertionError: (nan, nan, nan, nan)
>
>
>     This failure is specific to the 32 bit build and only appears when I run
>     'python -c"import scipy;scipy.test()"', but not in 'python -c"import
>     scipy.special;scipy.special.test()"'. In contrast to the ndimage
>     failure, this test fails for scipy.test() even if I remove all tests but
>     the scipy special tests.
>
>
> That's strange. Is the same function getting imported in both cases?

AFAICT the airye functions are the same in both cases (same type, repr, 
and __doc__). I reran the tests with the python -v switch and noticed 
that besides numpy and scipy, the matplotlib, PIL, sympy, zope, scikits, 
and entought packages are imported (why?). Maybe there's some 
monkeypatching or name shadowing going on? Anyway, I could not reproduce 
this failure on a clean installation of just python, numpy, scipy, and 
nose. The test_twodiags (test_linsolve.TestLinsolve) error and the 
test_linsolve.TestSplu.test_spilu_smoketest failure also disappear on 
this clean installation.

--
Christoph