Iterating over files of a huge directory
Oscar Benjamin
oscar.j.benjamin at gmail.com
Mon Dec 17 10:52:19 EST 2012
On 17 December 2012 15:28, Gilles Lenfant <gilles.lenfant at gmail.com> wrote:
> I have googled but did not find an efficient solution to my problem. My customer provides a directory with a huuuuge list of files (flat, potentially 100000+) and I cannot reasonably use os.listdir(this_path) unless creating a big memory footprint.
>
> So I'm looking for an iterator that yields the file names of a directory and does not make a giant list of what's in.
>
> i.e :
>
> for filename in enumerate_files(some_directory):
> # My cooking...
In the last couple of months there has been a lot of discussion (on
python-list or python-dev - not sure) about creating a library to more
efficiently iterate over the files in a directory. The result so far
is this library on github:
https://github.com/benhoyt/betterwalk
It says there that
"""
Somewhat relatedly, many people have also asked for a version of
os.listdir() that yields filenames as it iterates instead of returning
them as one big list.
So as well as a faster walk(), BetterWalk adds iterdir_stat() and
iterdir(). They're pretty easy to use, but see below for the full API
docs.
"""
Does that code work for you? If so, I imagine the author would be
interested to get some feedback on how well it works.
Alternatively, perhaps consider calling an external utility.
Oscar
More information about the Python-list
mailing list