[Numpy-discussion] Mysterious test_pareto failure on Travis

[Nathaniel Smith]

On Tue, Sep 4, 2012 at 9:17 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
> On Tue, Sep 4, 2012 at 12:41 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>> On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>>> On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>>> The last two Travis builds of master have failed consistently with the
>>>> same error:
>>>>   http://travis-ci.org/#!/numpy/numpy/builds
>>>> It looks like a real failure -- we're getting the same error on every
>>>> build variant, some sort of problem in test_pareto. Example:
>>>>   http://travis-ci.org/#!/numpy/numpy/jobs/2328823
>>>>
>>>> The obvious culprit would be the previous commit, which regenerated
>>>> mtrand.c with Cython 0.17:
>>>>   http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606
>>>>
>>>> What's weird, though, is that that commit passed just fine on Travis:
>>>>   http://travis-ci.org/#!/numpy/numpy/builds/2313124
>>>>
>>>> It's just the two commits since then that failed. But these commits
>>>> have been 1-line docstring changes, so I don't see how they could have
>>>> possibly created the problem.
>>>>
>>>> Also, the test passes fine with python 2.7 on my laptop with current master.
>>>>
>>>> Can anyone reproduce this failure? Any ideas what might be going on?
>>>
>>> I made this:
>>>
>>> https://github.com/numpy/numpy/issues/424
>>>
>>> It was me who updated the Cython file. It seemed to be working. I've
>>> added the issue
>>> to the release TODO.
>>
>> Ok, here is how to reproduce the problem:
>>
>> 1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
>> in the README:
>>
>> https://github.com/certik/numpy-vendor
>>
>> 2) run tests, you'll get:
>>
>> https://gist.github.com/3625509
>
> So the problem was actually introduced much earlier. Most probably it
> has never worked
> in 32bit Ubuntu 12.04. I tried for example this old commit:
>
> https://github.com/numpy/numpy/commit/3882d65c42acf6d5fff8cc9b3f410bb3e49c8af8
>
> and it still fails:
[...]
> But in any case, this seems to be a problem with the actual 32bit
> Ubuntu 12.04 itself. So maybe something in gcc
> has changed that now triggers the problem.

To be clear, the mismatching value is:

> /home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto()
-> np.testing.assert_array_almost_equal(actual, desired, decimal=15)
(Pdb) actual[1, 0]
52828779.702948704
(Pdb) desired[1, 0]
52828779.702948518

So they do match to 14 decimal points, and it's entirely possible that
this is just a problem of the test being too stringent in requiring 15
decimal points of match. Maybe the 32-bit GCC is spilling registers
differently, tripping the famous x86 idiosyncrasy where register
spilling triggers rounding. But I'd feel better if someone familiar
with the pareto code could confirm.

-n