Iterating over files of a huge directory
marduk
marduk at python.net
Mon Dec 17 10:50:21 EST 2012
On Mon, Dec 17, 2012, at 10:28 AM, Gilles Lenfant wrote:
> Hi,
>
> I have googled but did not find an efficient solution to my problem. My
> customer provides a directory with a huuuuge list of files (flat,
> potentially 100000+) and I cannot reasonably use os.listdir(this_path)
> unless creating a big memory footprint.
>
> So I'm looking for an iterator that yields the file names of a directory
> and does not make a giant list of what's in.
>
> i.e :
>
> for filename in enumerate_files(some_directory):
> # My cooking...
>
You could try using opendir[1] which is a binding to the posix call. I
believe that it returns an iterator (file-like) of the entries in the
directory.
[1] http://pypi.python.org/pypi/opendir/
More information about the Python-list
mailing list