[Numpy-discussion] I/O documentation and code

[Robert Kern]

On Sat, Jun 20, 2009 at 16:33, Ralf Gommers <ralf.gommers at googlemail.com> wrote:
>
> Hi,
>
> I'm working on the I/O documentation, and have a bunch of questions.
>
> 1. The npy/npz formats are documented in lib.format and in the NEP (http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt). Is lib.format the right place to add relevant parts of the NEP, or would doc.io be better?

What parts?

> Or create a separate page (maybe doc.npy_format)?

Probably all of the implemented NEPs should have their own place in
the documentation and other parts should reference the NEPs for
technical detail.

> And is the .npz format fixed or still in flux?

It's not as formalized as the .npy format, but I expect it to be at
least as solid as other code in numpy.

> 2. Is the .npy format version number (now at 1.0) independent of the numpy version numbering, when is it incremented, and will it be backwards compatible?

It is independent of numpy version numbering. If we do upgrade the
format, the code in numpy.io will still be able to read and write 1.0
files.

> 4. This page http://www.scipy.org/Data_sets_and_examples talks about including data sets with scipy, has this happened? Would it be possible to include a single small dataset in numpy for use in examples?

I think the dataset convention is entirely independent of numpy per
se. The current version of this stuff is in the scikits.learn package:

http://svn.scipy.org/svn/scikits/trunk/learn/scikits/learn/datasets/

The proposal could be turned into an "informative" NEP, of course. It
needs to be updated, though (e.g. it talks about not needing to
combine masked arrays and record arrays, but this has already been
done with the numpy.ma rewrite).

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco