[Python-Dev] PEP 446 (make FD non inheritable) ready for a final review

[Victor Stinner]

Hi,

I will try to answer to your worries. Tell me if I should complete the
PEP with these answers.

2013/8/23 Charles-François Natali <cf.natali at gmail.com>:
> Why does dup2() create inheritable FD, and not dup()?

Ah yes, there were as section explaining it. In a previous version of
the PEP (and its implementation), os.dup() and os.dup2() created
non-inheritable FD, but inheritable FD for standard steams (fd 0, 1,
2).

I did a research on http://code.ohloh.net/ to see how os.dup2() is
used in Python projects. 99.4% (169 projects on 170) uses os.dup2() to
replace standard streams: file descriptors 0, 1, 2 (stdin, stdout,
stderr); sometimes only stdout or stderr. I only found a short demo
script using dup2() with arbitrary file descriptor numbers (10 and 11)
to keep a copy of stdout and stderr before replacing them (also with
dup2).

I didn't find use cases of dup() to inherit file descriptors in the
Python standard library. It's the opposite: when os.dup() (or the C
function dup() is used), the FD must not be inherited. For example,
os.listdir(fd) duplicates fd: a child process must not inherit the
duplicated file descriptor, it may lead a security vulnerability (ex:
parent process running as root, whereas the child process is running
as a different used and not allowed to open the directory).


> For example, a lot of code uses the guarantee that dup()/open()...
> returns the lowest numbered file descriptor available, so code like
> this:
>
> r, w = os.pipe()
> if os.fork() == 0:
>     # child
>     os.close(r)
>     os.close(1)
>     dup(w)
>
> *will break*
>
> And that's a lot of code (e.g. that's what _posixsubprocess.c uses,
> but since it's implemented in C it's wouldn't be affected).

Yes, it will break. As I wrote in my previous email, we cannot solve
all issues listed in the Rationale section of the PEP without breaking
applications (or at least breaking backward compatibility). It is even
explicitly said in the "Backward Compatibility" section:
http://www.python.org/dev/peps/pep-0446/#backward-compatibility
"This PEP break applications relying on inheritance of file descriptors."

But I also added a hint to fix applications:
"Developers are encouraged to reuse the high-level Python module
subprocess which handles the inheritance of file descriptors in a
portable way."

If you don't want to use subprocess, yes, you will have to add
"os.set_inheritable(w)" in the child process.

About your example: I'm not sure that it is reliable/portable. I saw
daemon libraries closing *all* file descriptors and then expecting new
file descriptors to become 0, 1 and 2. Your example is different
because w is still open. On Windows, I have seen cases with only fd 0,
1, 2 open, and the next open() call gives the fd 10 or 13...

I'm optimistic and I expect that most Python applications and
libraries already use the subprocess module. The subprocess module
closes all file descriptors (except 0, 1, 2) since Python 3.2.
Developers relying on the FD inheritance and using the subprocess with
Python 3.2 or later already had to use the pass_fds parameter.


> Furthermore, many people use Python for system programming, and this
> change would be highly surprising.

Yes, it is a voluntary design choice (of the PEP). It is also said
explicitly in the "Backward Compatibility" section:
"Python does no more conform to POSIX, since file descriptors are now
made non-inheritable by default. Python was not designed to conform to
POSIX, but was designed to develop portable applications."

> So no matter what the final decision on this PEP is, it must be kept in mind.

The purpose of the PEP is to explain correctly the context and the
consequences of the changes, so Guido van Rossum can uses the PEP to
make its final decision.


>> The programming languages Go, Perl and Ruby make newly created file descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0 (1987) and Ruby 2.0 (2013).
>
> OK, but do they expose OS file descriptors?

Yes:

- Perl: fileno() function
- Ruby: fileno() method of a file object
- Go: fd() method of a file object


> Last time, I said that to me, the FD inheritance issue is solved on
> POSIX by the subprocess module which passes close_fds. In my own code,
> I use subprocess, which is the "official", portable and safe way to
> create child processes in Python. Someone using fork() + exec() should
> know what he's doing, and be able to deal with the consequences: I'm
> not only talking about FD inheritance, but also about
> async-signal/multi-threaded safety ;-)

The subprocess module has still a (minor?) race condition in the child
process. Another C thread can create a new file descriptor after the
subprocess module closed all file descriptors and before exec(). I
hope that it is very unlikely, but it can happen. It's also explained
in the PEP (see "Closing All Open File Descriptors"). I suppose that
the race condition explains why Linux still has no closefrom() or
nextfd() system calls. IMO the kernel is the best place to decide
which FD should be kept, and which must not be inherited (must be
closed), in the child process. I like the close-on-exec flag (and
HANDLE_FLAG_INHERIT on Windows).


> As for Windows, since it doesn't have fork(), it would make sense to
> make its FD non heritable by default.

As said in the "Inheritance of File Descriptors on Windows" section,
Python gets inheritable handles and file descriptors because it does
not use the Windows native API. Applications developed for Windows
using the native API only create non-inheritable handles, and so don't
have all these annoying inheritance issues.

Windows does not have a fork() function, but handles and file
descriptors can be inherited using CreateProcess() and spawn(), see
the table in the "Status of Python 3.3" section.


> And then use what you describe
> here to selectively inherit FDs (i.e. implement keep_fds):
> """
> Since Windows Vista, CreateProcess() supports an extension of the
> STARTUPINFO struture: the STARTUPINFOEX structure. Using this new
> structure, it is possible to specify a list of handles to inherit:
> PROC_THREAD_ATTRIBUTE_HANDLE_LIST. Read Programmatically controlling
> which handles are inherited by new processes in Win32 (Raymond Chen,
> Dec 2011) for more information.
> """

This feature can only be used to inherit *handles*, and it does not
need of an intermediate program. If you want to inherit only some file
descriptors, you need an intermediate program which will recreates
them and then use spawn() (or directly the reserved fields of the
STARTUPINFO structure). Using an intermediate program has unexpected
consequences, so I prefer to use this option.

The feature is also specific to Windows 7, a recent Windows version,
whereas there are still millions of Windows XP installations (and I
read that Python 3.4 will still supported Windows XP!).

We need more feedback from users to know their use cases before
chosing the best implementation of pass_handles/pass_fds on Windows.

Richard agreed that this point can be deferred.

Victor