[Numpy-discussion] numpy.load raising IOError but EOFError expected

[Anne Archibald]

On 28 June 2010 10:52, Ruben Salvador <rsalvador.wk at gmail.com> wrote:
> Sorry I had no access during these days.
>
> Thanks for the answer Friedrich, I had already checked numpy.savez, but
> unfortunately I cannot make use of it. I don't have all the data needed to
> be saved at the same time...it is produced each time I run a test.

I think people are uncomfortable because .npy files are not designed
to contain more than one array. It's a bit like concatenating a whole
lot of .PNG files together - while a good decoder could pick them
apart again, it's a highly misleading file since the headers do not
contain any information about all the other files. npy files are
similarly self-describing, and so concatenating them is a peculiar
sort of thing to do. Why not simply save a separate file each time, so
that you have a directory full of files? Or, if you must have just one
file, use np.savez (loading the old one each time then saving the
expanded object).

Come to think of it, it's possible to "append" files to an existing
zip file without rewriting the whole thing. Does numpy.savez allow
this mode?

That said, good exception hygiene argues that np.load should throw
EOFErrors rather than the more generic IOErrors, but I don't know how
difficult this would be to achieve.


Anne

> Thanks anyway!
>
> Any other idea why this is happening? Is it expected behavior?
>
> On Thu, Jun 24, 2010 at 7:30 PM, Friedrich Romstedt
> <friedrichromstedt at gmail.com> wrote:
>>
>> 2010/6/23 Ruben Salvador <rsalvador.wk at gmail.com>:
>> > Therefore, is this a bug? Shouldn't EOFError be raised instead of
>> > IOError?
>> > Or am I missunderstanding something? If this is not a bug, how can I
>> > detect
>> > the EOF to stop reading (I expect a way for this to work without
>> > tweaking
>> > the code with saving first in the file the number of dumps done)?
>>
>> Maybe you can make use of numpy.savez,
>>
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html#numpy-savez
>> .
>>
>> Friedrich
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> --
> Rubén Salvador
> PhD student @ Centro de Electrónica Industrial (CEI)
> http://www.cei.upm.es
> Blog: http://aesatcei.wordpress.com
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>