[SciPy-User] Maximum file size for .npz format?

[Anne Archibald]

On 11 March 2010 10:48, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <jgomezdans at gmail.com> wrote:
>> Hi!
>> I need to save a fairly large set of arrays to disk. I have saved it using
>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>> large ;D). When I try to load it using numpy.load, the zipfile module
>> compains about
>> BadZipfile: Bad magic number for file header
>>
>> I can't open it with the normal zip utility present on the system, but it
>> could be that it's barfing about files being larger than 2Gb.
>> Is there some file limit for npzs?
>
> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> not yet support the ZIP64 format.

Could it be arranged that an exception is raised when creating a >4GB
.npz file, so people do not find themselves with unrecoverable data?

>> Is there anyway I can recover the data (I
>> guess I could try decompressing the file with 7z and extracting the
>> individual npy files?)
>
> Possibly. However, if the normal zip utility isn't working, 7z
> probably won't, either. Worth a try, though.

If your data is valuable enough — irreplaceable space mission results,
say — some careful spelunking in the code combined with some knowledge
about your data might allow you to semi-manually reconstruct it from
the damaged zip file. This would be a long and painful labour, but
would probably produce a python module that supported zip64 as an
incidental side effect.

Anne

> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>