[SciPy-User] Maximum file size for .npz format?

[Lafras Uys]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> I need to save a fairly large set of arrays to disk. I have saved it using
>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>> large ;D). When I try to load it using numpy.load, the zipfile module
>> compains about
>> BadZipfile: Bad magic number for file header
>>
>> I can't open it with the normal zip utility present on the system, but it
>> could be that it's barfing about files being larger than 2Gb.
>> Is there some file limit for npzs?
> 
> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> not yet support the ZIP64 format.
> 
>> Is there anyway I can recover the data (I
>> guess I could try decompressing the file with 7z and extracting the
>> individual npy files?)
> 
> Possibly. However, if the normal zip utility isn't working, 7z
> probably won't, either. Worth a try, though.

I've had similar problems, my solution was to move to HDF5. There are
two options for accessing and working with HDF files from python: h5py
(http://code.google.com/p/h5py/) and pytables
(http://www.pytables.org/). Both packages have built in numpy support.

Regards,

Lafras
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuZ8jcACgkQKUpCd+bV+kruKwCghfG0yAo/eRXzDZxH6i1eOyfn
bnUAoLsuLB2O9qyvJV7CP3jXT9OcMwye
=1xc3
-----END PGP SIGNATURE-----