[MATRIX-SIG] Possible bug in Numeric LoadArray function

[Jim Hugunin]

Bill White wrote:
> 
> > > A pickled file for an integer array was created on an SGI Onyx.
> > > This machine has 4 byte ints and 4 byte longs.
> > >
> > > The picked file would not read on a DEC alpha (64 bit longs).
> > > The reshape function in LoadArray failed because the
> > > byte counts didn't match.
> >
> > I am not surprised - the pickling approach used in NumPy is not really
> > portable. Anyway, it will have to be rewritten to profit from cPickle
> > under Python 1.5. Unfortunately, there is no perfect solution; either
> > pickling must make assumptions about the binary format of its data
> > types (as it does now), or it must apply a conversion, which can
> > become very time consuming for large arrays.
> 
> Well, forgive me for saying the obvious thing, but perhaps the best
> approach is to do the following, assuming that people will be moving
> mostly to similar or identical architectures, but with the occasional
> radical excursion.

<Proposed solution snipped>

This is in fact the assumption made by the current Pickling algorithm. 
It includes the actual number of bytes used to store the particular
numeric type when it does the pickle.  On platforms where this number
matches the native size, no conversions are necessary.  The idea always
was to add code to LoadArray that would check this assumption and when
moving to a platform with different numeric sizes it would then (and
only then) perform a potentially expensive conversion operation.

I made sure that DumpArray put this information into the pickle format,
so that in the future somebody with the time could write a correct
implementation of LoadArray without needing to make any incompatible
changes to the array pickling format.  Anybody want to actually write
the code?

-Jim

_______________
MATRIX-SIG  - SIG on Matrix Math for Python

send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________