[Numpy-discussion] Mysterious test_pareto failure on Travis

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 1:48 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Sep 4, 2012 at 4:37 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Tue, Sep 4, 2012 at 9:17 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>>> On Tue, Sep 4, 2012 at 12:41 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>>>> On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>>>>> On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>>>>> The last two Travis builds of master have failed consistently with the
>>>>>> same error:
>>>>>>   http://travis-ci.org/#!/numpy/numpy/builds
>>>>>> It looks like a real failure -- we're getting the same error on every
>>>>>> build variant, some sort of problem in test_pareto. Example:
>>>>>>   http://travis-ci.org/#!/numpy/numpy/jobs/2328823
>>>>>>
>>>>>> The obvious culprit would be the previous commit, which regenerated
>>>>>> mtrand.c with Cython 0.17:
>>>>>>   http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606
>>>>>>
>>>>>> What's weird, though, is that that commit passed just fine on Travis:
>>>>>>   http://travis-ci.org/#!/numpy/numpy/builds/2313124
>>>>>>
>>>>>> It's just the two commits since then that failed. But these commits
>>>>>> have been 1-line docstring changes, so I don't see how they could have
>>>>>> possibly created the problem.
>>>>>>
>>>>>> Also, the test passes fine with python 2.7 on my laptop with current master.
>>>>>>
>>>>>> Can anyone reproduce this failure? Any ideas what might be going on?
>>>>>
>>>>> I made this:
>>>>>
>>>>> https://github.com/numpy/numpy/issues/424
>>>>>
>>>>> It was me who updated the Cython file. It seemed to be working. I've
>>>>> added the issue
>>>>> to the release TODO.
>>>>
>>>> Ok, here is how to reproduce the problem:
>>>>
>>>> 1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
>>>> in the README:
>>>>
>>>> https://github.com/certik/numpy-vendor
>>>>
>>>> 2) run tests, you'll get:
>>>>
>>>> https://gist.github.com/3625509
>>>
>>> So the problem was actually introduced much earlier. Most probably it
>>> has never worked
>>> in 32bit Ubuntu 12.04. I tried for example this old commit:
>>>
>>> https://github.com/numpy/numpy/commit/3882d65c42acf6d5fff8cc9b3f410bb3e49c8af8
>>>
>>> and it still fails:
>> [...]
>>> But in any case, this seems to be a problem with the actual 32bit
>>> Ubuntu 12.04 itself. So maybe something in gcc
>>> has changed that now triggers the problem.
>>
>> To be clear, the mismatching value is:
>>
>>> /home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto()
>> -> np.testing.assert_array_almost_equal(actual, desired, decimal=15)
>> (Pdb) actual[1, 0]
>> 52828779.702948704
>> (Pdb) desired[1, 0]
>> 52828779.702948518
>>
>> So they do match to 14 decimal points, and it's entirely possible that
>> this is just a problem of the test being too stringent in requiring 15
>> decimal points of match. Maybe the 32-bit GCC is spilling registers
>> differently, tripping the famous x86 idiosyncrasy where register
>> spilling triggers rounding. But I'd feel better if someone familiar
>> with the pareto code could confirm.
>
> I don't understand this. Isn't assert_almost_equal testing decimals
> not significant digits?

Yes it is, see the docs:

    The test is equivalent to ``abs(desired-actual) < 0.5 * 10**(-decimal)``.

and the unit test is badly written, because that floating point
number has maybe 14 or 15 good significant digits, but it simply does not have
15 digits after the decimal point, so in particular, that test is
testing that the two
floating point numbers are exactly equal.


Here is a fix:

https://github.com/numpy/numpy/pull/425

Let me know if it is ok to merge it, so that we can work on other
issues and have a working test suite.

Ondrej