[ python-Bugs-1519452 ] zipfile -- too many files?

[SourceForge.net]

Bugs item #1519452, was opened at 2006-07-09 05:57
Message generated for change (Settings changed) made by gbrandl
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1519452&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
>Status: Closed
Resolution: Fixed
Priority: 5
Submitted By: Joe Brown (shmengie)
Assigned to: Nobody/Anonymous (nobody)
Summary: zipfile -- too many files?

Initial Comment:
While creating a huge zipfile (probably too big) the
following error appeared.

Exception exceptions.OverflowError: 'long int too large
to convert to int' in <bound method ZipFile.__del__ of
<zipfile.ZipFile instance at 0x009BD198>> ignored

I assume this is during the closing of the zipfile,
further assuming too many files were added to the archive.

----------------------------------------------------------------------

Comment By: Ronald Oussoren (ronaldoussoren)
Date: 2006-07-13 08:17

Message:
Logged In: YES 
user_id=580910

hmm, I should read all new comments before responding :-(

To use the ZIP64 extensions with python2.5 you must add 'allowZip64=True' to 
arguments of the zipfile constructor. ZIP64 is disabled by default because the 
unix zip tools don't support zip64.

----------------------------------------------------------------------

Comment By: Ronald Oussoren (ronaldoussoren)
Date: 2006-07-13 07:47

Message:
Logged In: YES 
user_id=580910

The problem with zipfile is not the number of files, but the size of the archive. 
The basic zip file format stores file sizes as 32 bit (signed) integers, there is an 
extension for using 64 bit integers, but that isn't supported in python 2.4.

----------------------------------------------------------------------

Comment By: Joe Brown (shmengie)
Date: 2006-07-12 02:17

Message:
Logged In: YES 
user_id=729883

Windows 2.5 beta 2 responded like this:

Traceback (most recent call last):
  File "C:\python_scripts\archive.py", line 27, in <module>
    main()
  File "C:\python_scripts\archive.py", line 23, in main
    zipper(zip, folder)
  File "C:\python_scripts\archive.py", line 15, in zipper
    zip.write(os.path.join(root, item))
  File "C:\Python25\lib\zipfile.py", line 561, in write
    self._writecheck(zinfo)
  File "C:\Python25\lib\zipfile.py", line 536, in _writecheck
    raise LargeZipFile("Zipfile size would require ZIP64
extensions")
zipfile.LargeZipFile: Zipfile size would require ZIP64
extensions
C:\Python25\lib\struct.py:63: DeprecationWarning: struct
integer overflow maskin
g is deprecated
  return o.pack(*args)

----------------------------------------------------------------------

Comment By: Joe Brown (shmengie)
Date: 2006-07-11 20:08

Message:
Logged In: YES 
user_id=729883

The zip file was on order of 6-9 GBs, surely over 32000
files.  I will give Python2.5 a shot and see how that works.

I've stumbled on another bug with tarfile.  It's not
reporting an error, but I only find 350 files when I
unarchive.  That's entirely different bug tho :-o

----------------------------------------------------------------------

Comment By: Ronald Oussoren (ronaldoussoren)
Date: 2006-07-11 17:47

Message:
Logged In: YES 
user_id=580910

How large is the resulting file? Zipfile in python2.4 cannot handle zipfiles that 
require the zip64 extensions and fails misserably when you do cross that limit.

This was fixed in python 2.5.

----------------------------------------------------------------------

Comment By: Joe Brown (shmengie)
Date: 2006-07-09 06:01

Message:
Logged In: YES 
user_id=729883

Platform: Windows SBS 2003
Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit
(Intel)] on win32

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1519452&group_id=5470