[Numpy-discussion] ENH IncrementalWriter for .npy files

[Gabor Kovacs]

Dear All,

I would like to add a class for writing one (possibly big) .npy file
saving multiple (same dtype, compatible shape) arrays. My use case was
the saving of slowly accumulating data regularly for a long time into
one file.

Please find a first implementation under
https://github.com/numpy/numpy/pull/4987 . It currently supports
writing a new file only and only in C order in the file. Opening an
existing file for append and reading back parts from a very big .npy
file would be straightforward next steps for a full featured class.

The .npy file format is only affected by leaving some extra space for
re-writing the header later with a possibly bigger "shape" field,
respecting the 16-byte alignment.

Example:
```
A=np.array([[0,1,2,3,4,5,6,7],[8,9,10,11,12,13,14,15]])
with np.IncrementalWriter("testfile.npy",hdrupdate=True,flush=True) as W:
    W.save(A)
    W.save(A)
```

Feel free to comment this idea.

Cheers,
Gabor