[Python-Dev] PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)
Ben Hoyt
benhoyt at gmail.com
Tue Jul 1 14:26:15 CEST 2014
Thanks, Victor.
I don't have any experience with dir_fd handling, so unfortunately
can't really comment here.
What advantages does it bring? I notice that even os.listdir() on
Python 3.4 doesn't have anything related to file descriptors, so I'd
be in favour of not including support. We can always add it later.
-Ben
On Tue, Jul 1, 2014 at 3:44 AM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> IMO we must decide if scandir() must support or not file descriptor.
> It's an important decision which has an important impact on the API.
>
>
> To support scandir(fd), the minimum is to store dir_fd in DirEntry:
> dir_fd would be None for scandir(str).
>
>
> scandir(fd) must not close the file descriptor, it should be done by
> the caller. Handling the lifetime of the file descriptor is a
> difficult problem, it's better to let the user decide how to handle
> it.
>
> There is the problem of the limit of open file descriptors, usually
> 1024 but it can be lower. It *can* be an issue for very deep file
> hierarchy.
>
> If we choose to support scandir(fd), it's probably safer to not use
> scandir(fd) by default in os.walk() (use scandir(str) instead), wait
> until the feature is well tested, corner cases are well known, etc.
>
>
> The second step is to enhance pathlib.Path to support an optional file
> descriptor. Path already has methods on filenames like chmod(),
> exists(), rename(), etc.
>
>
> Example:
>
> fd = os.open(path, os.O_DIRECTORY)
> try:
> for entry in os.scandir(fd):
> # ... use entry to benefit of entry cache: is_dir(), lstat_result ...
> path = pathlib.Path(entry.name, dir_fd=entry.dir_fd)
> # ... use path which uses dir_fd ...
> finally:
> os.close(fd)
>
> Problem: if the path object is stored somewhere and use after the
> loop, Path methods will fail because dir_fd was closed. It's even
> worse if a new directory uses the same file descriptor :-/ (security
> issue, or at least tricky bugs!)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
More information about the Python-Dev
mailing list