[SciPy-user] Changes in SVN scipy_core

[Travis Oliphant]

Hi Folks,

It's great to see the configuration stuff getting better in 
scipy.distutils.  Thanks to Pearu for all his hard work on 
scipy.distutils. 

I want to let you all know about the recent changes in the SVN version 
of scipy_core.  As you may recall, I dramatically improved the way the 
data in an array can be understood by improving the PyArray_Descr 
structure and making it an object.  As part of that improvement, it 
became clear that the NOTSWAPPED flag was an anomaly and shouldn't be a 
flag on the array itself.  The byte-order of the data is a property of 
the data-type descriptor.  This is especially clear when considering 
records which according to the array protocol can have some fields in 
one byte order and some in another.

As a result, I've removed the NOTSWAPPED flag from the array object.  
The equivalent of arr.flags.notswapped is now arr.dtypedescr.isnative.  
In C, the macro PyArray_ISNOTSWAPPED(arr) is still available but it 
checks the data-type descriptor. All C-API and python calls that used to 
pass a swap paramter along with a data-type descriptor have the swap 
paramter deleted.

These C-API changes mean you will need to fully rebuild (remove the 
build directory and build) both scipy_core and scipy if you check out 
the new SVN version.  I realize that the pace of development has been 
rapid --- but I'm on a tight schedule ;-)   Hopefully, this wil be the 
last relatively major change to the code base for a while.

All tests pass for me for full scipy after these changes.  As we all 
know, however, there still may be remaining issues.  One issue, for 
example, is that a.copy() returns a copy of the data (with the same 
data-type descriptor so no change in the byte-order is done).

There is a new method

a.newbyteorder(<arg>) 

which returns the equivalent of

a.view(a.dtypedescr.newbyteorder(<arg>)) 

which returns a new view of the data using a different byte-order 
description.

Note that the newbyteorder method of a dtypedescr object returns a new 
copy of the dtypedescr object with the byte-orders swapped if <arg> is 
not given or forces the byteorder to a particular value if arg is given 
(and not arg=='|' which means ignore). All fields of the dtypedescr 
object are (recursively) updated with the same rule as well.

Look what comes along for the ride because of these changes:

a = ones((5,), '<i4')  # define a little-endian array
b = ones((5,), '>i4')  # define a big-endian array

a.tostring()
'\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00'
b.tostring()
'\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x01'

You can use these byte-order flags anywhere a data-type descriptor 
(dtype parameter) is required.  I have not tested all the possibilities, 
of course, so there may still be outstanding issues. 

Note, however, that if you work with arrays that are not in native-byte 
order some operations will generally be slower.

Regards,

-Travis