[Numpy-discussion] Writing/reading a numeric array to a file

[Todd Miller]

N. Volbers wrote:

> Hello everyone,
>
> I have been using Numeric and/or numarray for a while now and I now I 
> have a question concerning the reading and writing data to a file.
>
> Until now I had used pycdf, which is an interface to the netcdf 
> library, to save and load my numeric arrays.  However, since my 
> application already has too many dependencies, I would like to cut 
> down on these two and replace the saving and loading by a method 
> intrinisc to Numeric or numarray.
> Pickling is not an alternative to me, because I need to have a 
> representation that can be read by other programs (and yet is faster 
> than ASCII). NetCDF was a good choice for that, but I would really 
> like to not depend on it.
>
> So my questions are:
>
> (1) Are the numarray functions 'fromfile' and 'tofile' 100% portable 
> among platforms, i.e. do they automatically recognize endian-ness and 
> such?

No.   But depending on how general a solution you need here,  this can 
be fairly easy (i.e. for numerical arrays only).

> (2) Does it make sense to still use numarray? 

Absolutely...  numarray works with records and memory mapping now.   But 
you also need to keep a shrewd eye on scipy newcore and recognize that 
it will most probably replace numarray over the course of the next year 
or two as it becomes sufficiently complete and stable to do so.

There are smart things to do now if you want to use numarray:

a. Use the Numeric-compatible C-API as much as possible.

b. Keep an eye on the introduction of newcore compatible typenames 
(int32 vs Int32), keywords (dtype vs. type), and attributes and use 
those as you write new code in numarray.

c. Use the array protocol.

> I know Travis would say "use scipy_core". However, for me this would 
> provide much unneeded functionality and I have not yet found an easy 
> way to install scipy_core (it seems to require ATLAS and such, which 
> are not so easy to install if you don't have it prepackaged).  And 
> after all, my goal is to cut down dependencies...
>
> (3) Let me restate question (2): Will numarray still be maintained?

numarray will be maintained at STScI until (a) newcore is ready to 
replace it or (b) our budget gets cut to the point that we cannot and no 
one else is interested.  Neither of those is guaranteed to happen,  but 
(a) looks likely to us.  STScI has the same problems with installation 
and dependencies so they'll have to be solved before we use newcore either.

> Or is it also deprecated?  What would you advice someone who just 
> needs the array interface?

Pay careful attention to the __array_struct__ attribute described here: 

http://numeric.scipy.org/array_interface.html

It's the easiest and best performing method to interface from C.  To 
interface from Python you have to use more of the protocol.

> And of course,
>
> (4) What solutions do you use to save/load data to files?

numarray was written to support astronomical data processing.  The 
dominant data format in astronomy is called FITS.  STScI has another 
package called PyFITS which is built on numarray and exposes the FITS 
format to Python.

Regards,
Todd