How to process a very large (4Gb) tarfile from python?
Terry Carroll
carroll at nospam-tjc.com
Thu Jul 17 11:55:06 EDT 2008
On Thu, 17 Jul 2008 06:14:45 -0700 (PDT), Uwe Schmitt
<rocksportrocker at googlemail.com> wrote:
>I had a look at tarfile.py in my current Python 2.5 installations
>lib path. The iterator caches TarInfo objects in a list
>tf.members . If you only want to iterate and you are not interested
>in more functionallity, you could use "tf.members=[]" inside
>your loop. This is a dirty hack !
Thanks, Uwe. That works fine for me. It now reads through all 2.5
million members, in about 30 minutes, never going above a 4M working
set.
More information about the Python-list
mailing list