how can I put a 1Gb file in a zipfile??

[Bengt Richter]

On Sun, 20 Mar 2005 14:17:20 -0600, Jeff Epler <jepler at unpythonic.net> wrote:

>
>--liOOAslEiF7prFVr
>Content-Type: text/plain; charset=us-ascii
>Content-Disposition: inline
>Content-Transfer-Encoding: quoted-printable
>
>The limits of ZIP files according to the folks who make info-zip:
>http://www.info-zip.org/pub/infozip/FAQ.html#limits
>
> statistic                             limit
>number of files                        65,536
>uncompressed size of a single file       4 GB
>compressed size of a single file         4 GB
>total size of archive                  256 TB
>maximum path/filename length            64 KB=20
>
>I had no trouble creating a zip file from a 4GB file filled with '\0'
>bytes:
>$ python bennie.py
>$ ls -ls test.zip big
>  12 -rw-rw-r--  1 jepler jepler 4294967296 Mar 20 14:11 big
>4084 -rw-rw-r--  1 jepler jepler    4174545 Mar 20 14:14 test.zip
>
>I'm using Python 2.3.3 on Fedora Core 2.
>#------------------------------------------------------------------------
># bennie.py
>def make_4gb_file(f):
>    f =3D open(f, "w")
>    f.seek ( 4 * 1024 * 1024 * 1024 - 1)
>    f.write("\0")
>    f.close()
>
Not quite OT[?]:
This makes me think there ought to be a way of making at least python's builtin open see virtual file system objects,
analogous to StringIO creating file objects.

If we had a mountvfs('/some/unix/and/or/maybe/win/style/path/vfs', vfsclass(some, args, if_desired))
which would result in that open(/some/unix/and/or/maybe/win/style/path/vfs/morepath/filename.ext', mode)
(where vfsclass(some, args, if_desired) => vsfclass_intance) would call
vfsclass_instance.open('morepath.filename.ext', mode) which could then return an object that could support
file operations like returning 4gb of virtually read-by-read-method data, or otherwise acting like an open file
object of a real file system that python programs and library function using open and file would find
if given the mounted path. Subdirectories could be fixed for starters, but virtualizing subdirectory creation
etc would be possible if you intercepted the right interface calls and implemented it in the vfs.

This would let you define a virtual file in place of the real file above, and also would allow a lot of transparent
testing of file-using software that takes paths and file names, not open file objects.

Of course you can't affect what the underlying os sees without getting into its file system machinery, but
being able to mount virtual file systems into what os.open sees would cover a lot of ground ISTM. One
could argue pro and con about supporting virtual mounts into both unix and windows-style paths.

>import zipfile
>z =3D zipfile.ZipFile("/tmp/test.zip", "w", zipfile.ZIP_DEFLATED)
>make_4gb_file("/tmp/big")
>z.write("/tmp/big")
>z.close()
>#------------------------------------------------------------------------

Regards,
Bengt Richter