[Numpy-discussion] Bizarre errors with byteswapping, complex256, PPC

[Matthew Brett]

Hi,

On Wed, Jun 20, 2012 at 11:57 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, Jun 21, 2012 at 12:11 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Wed, Jun 20, 2012 at 10:43 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Wed, Jun 20, 2012 at 4:11 PM, Matthew Brett <matthew.brett at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Wed, Jun 20, 2012 at 3:05 PM, Charles R Harris
>> >> <charlesr.harris at gmail.com> wrote:
>> >> >
>> >> >
>> >> > On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett
>> >> > <matthew.brett at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant
>> >> >> <travis at continuum.io>
>> >> >> wrote:
>> >> >> > This looks like a problem with comparisons of floating point
>> >> >> > numbers
>> >> >> > rather than a byteswapping problem per-say.   Try to use an almost
>> >> >> > equal
>> >> >> > comparison instead.
>> >> >>
>> >> >> Is that right - that the byteswapped versions might not be strictly
>> >> >> equal to identical numbers but not byteswapped?
>> >> >>
>> >> >> But I should maybe have been clearer - they also subtract wrongly:
>> >> >>
>> >> >> <script>
>> >> >> import numpy as np
>> >> >>
>> >> >> arr = np.arange(10, dtype=np.complex256)
>> >> >> bs_arr = arr.byteswap().newbyteorder('S')
>> >> >> print arr
>> >> >> print bs_arr
>> >> >> print arr - bs_arr
>> >> >> print arr - bs_arr
>> >> >> print arr - bs_arr
>> >> >> </script>
>> >> >>
>> >> >> (np-devel)[mb312 at joshlegacy ~/tmp]$ python funny_bs.py
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >>
>> >> >> (wrong)
>> >> >>
>> >> >> (np-devel)[mb312 at joshlegacy ~/tmp]$ python funny_bs.py
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >> >>  0.0+0.0j
>> >> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >> >>  0.0+0.0j
>> >> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >> >>  0.0+0.0j
>> >> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> >>
>> >> >> (right)
>> >> >>
>> >> >> See you,
>> >> >>
>> >> >
>> >> > Long doubles on PPC consist of two doubles, so I expect you need to
>> >> > swap
>> >> > both doubles instead of 16 bytes. Strictly speaking, numpy doesn't
>> >> > support
>> >> > non ieee floats.
>> >>
>> >> Well - the byteswapping appears to be correct in that the array is
>> >> displayed with the correct values, but then, when doing a subtraction
>> >> on the array, most of the time it is incorrect, but whether it is
>> >> correct or incorrect, appears to be random even with the same
>> >> variables and memory.
>> >>
>> >> Float128 and other numpy dtypes appear to be correct using the same
>> >> tests.
>> >
>> >
>> > Thinking about it, that makes sense because the swapped version is
>> > probably
>> > incorrect ;) That is, the PPC was (is?) selectable to run either little
>> > endian or big endian, so the real test would be if long doubles were
>> > portable between machines set up different ways. The only machines I
>> > know of
>> > are little endian, but IIRC, there was at least one brand that was big
>> > endian. However, I suspect we are just reversing the whole 16 bytes, so
>> > even
>> > though that is pretty much a meaningless thing to do, it should work...
>> >
>> > Is this something that only happens on 32 bit machines?
>>
>> The PPC machines I have record themselves as big endian (2 running OSX
>> and 1 running Debian wheezy).
>>
>> I can only get the Debian wheezy machine to misbehave in this way.
>>
>> The original report of the problem was on a POWER7 machine running
>> Debian - wheezy I think.  I guess this is 64 bit - Yarik - do you
>> know?
>>
>
> Looks like I got little/big endian reversed. Anyway, this is very strange.
> Same compilers on both machines?

gcc-4.6.3-1 on the machine I have access to.

The POWER7 machine appears to be running an older kernel, so my guess
is it has an older gcc too, but I don't have access I'm afraid.

>  I don't understand why this wouldn't show
> up on other machines with float128, and in particular why is should only
> happen for complex256 and Debian. What is the extended precision type in OSX
> on PPC?

OSX uses double pairs too - and I've been testing on them for a while
- so I was also surprised to see this when we got to Debian...

See you,

Matthew