[SciPy-User] Maximum file size for .npz format?

[Robert Kern]

On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <jgomezdans at gmail.com> wrote:
> Hi!
> I need to save a fairly large set of arrays to disk. I have saved it using
> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
> large ;D). When I try to load it using numpy.load, the zipfile module
> compains about
> BadZipfile: Bad magic number for file header
>
> I can't open it with the normal zip utility present on the system, but it
> could be that it's barfing about files being larger than 2Gb.
> Is there some file limit for npzs?

Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
not yet support the ZIP64 format.

> Is there anyway I can recover the data (I
> guess I could try decompressing the file with 7z and extracting the
> individual npy files?)

Possibly. However, if the normal zip utility isn't working, 7z
probably won't, either. Worth a try, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco