[Python-checkins] peps: PEP 446: cleanup
victor.stinner
python-checkins at python.org
Tue Aug 6 02:19:04 CEST 2013
http://hg.python.org/peps/rev/2797d53b1859
changeset: 5035:2797d53b1859
user: Victor Stinner <victor.stinner at gmail.com>
date: Tue Aug 06 02:18:49 2013 +0200
summary:
PEP 446: cleanup
files:
pep-0446.txt | 163 +++++++++++++++++++-------------------
1 files changed, 81 insertions(+), 82 deletions(-)
diff --git a/pep-0446.txt b/pep-0446.txt
--- a/pep-0446.txt
+++ b/pep-0446.txt
@@ -16,10 +16,9 @@
Leaking file descriptors in child processes causes various annoying
issues and is a known major security vulnerability. This PEP proposes to
make all file descriptors created by Python non-inheritable by default
-to have a well defined and portable behaviour and reduce the risk of
-these issues. This PEP fixes also a race condition
-in multithreaded applications on operating systems supporting atomic
-flags to create non-inheritable file descriptors.
+to reduces the risk of these issues. This PEP fixes also a race
+condition in multithreaded applications on operating systems supporting
+atomic flags to create non-inheritable file descriptors.
Rationale
@@ -30,16 +29,16 @@
Each operating system handles the inheritance of file descriptors
differently. Windows creates non-inheritable file descriptors by
-default, whereas UNIX creates inheritable file descriptors. Python
-prefers the POSIX API over the native Windows API to have a single code
-base, and so creates inheritable file descriptors.
+default, whereas UNIX creates inheritable file descriptors by default.
+Python prefers the POSIX API over the native Windows API to have a
+single code base, and so it creates inheritable file descriptors.
There is one exception: ``os.pipe()`` creates non-inheritable pipes on
-Windows, whereas it creates inheritable pipes on UNIX. The reason comes
-from an implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on
-Windows, whereas it calls ``pipe()`` on UNIX. The call to
-``CreatePipe()`` was added in 1994, before the introduction of
-``pipe()`` in the POSIX API in Windows 98. The `issue #4708
+Windows, whereas it creates inheritable pipes on UNIX. The reason is an
+implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows
+(native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call
+to ``CreatePipe()`` was added in Python in 1994, before the introduction
+of ``pipe()`` in the POSIX API in Windows 98. The `issue #4708
<http://bugs.python.org/issue4708>`_ proposes to change ``os.pipe()`` on
Windows to create inheritable pipes.
@@ -51,29 +50,29 @@
``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which
defines if a handle can be inherited in a child process or not. For the
POSIX API, the C runtime (CRT) provides also file descriptors (C type
-``int``). The handle of a file descriptor can be retrieved using
-``_get_osfhandle(fd)``. A file descriptor can be created from a handle
-using ``_open_osfhandle(handle)``.
+``int``). The handle of a file descriptor can be get using the
+function ``_get_osfhandle(fd)``. A file descriptor can be created from a
+handle using the function ``_open_osfhandle(handle)``.
-Handles are only inherited if their inheritable flag
+Using `CreateProcess()
+<http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_,
+handles are only inherited if their inheritable flag
(``HANDLE_FLAG_INHERIT``) is set and if the ``bInheritHandles``
-parameter of `CreateProcess()
-<http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_
-is ``TRUE``. Using ``CreateProcess()``, all file descriptors except
-standard streams (0, 1, 2) are closed in the child process, even if
-``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all
-inheritable file descriptors are inherited in the child process. This
-function uses the undocumented fields *cbReserved2* and *lpReserved2* of
-the `STARTUPINFO
+parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors
+except standard streams (0, 1, 2) are closed in the child process, even
+if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all
+inheritable handles and all inheritable file descriptors are inherited
+in the child process. This function uses the undocumented fields
+*cbReserved2* and *lpReserved2* of the `STARTUPINFO
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms686331%28v=vs.85%29.aspx>`_
structure to pass an array of file descriptors.
-To replace standard streams (stdin, stdout, stderr), the
-``STARTF_USESTDHANDLES`` flag must be set in the *dwFlags* field of the
-``STARTUPINFO`` structure and the *bInheritHandles* parameter of
-``CreateProcess()`` must be set to ``TRUE``. So when at least one
-standard stream is replaced, all inheritable handles are inherited by
-the child process.
+To replace standard streams (stdin, stdout, stderr) using
+``CreateProcess()``, the ``STARTF_USESTDHANDLES`` flag must be set in
+the *dwFlags* field of the ``STARTUPINFO`` structure and the
+*bInheritHandles* parameter of ``CreateProcess()`` must be set to
+``TRUE``. So when at least one standard stream is replaced, all
+inheritable handles are inherited by the child process.
See also:
@@ -88,8 +87,8 @@
POSIX provides a *close-on-exec* flag on file descriptors to close
automatically a file descriptor when the C function ``execv()`` is
-called. File descriptors with the *close-on-exec* flag unset are
-inherited in the child process, file descriptros with the flag set are
+called. File descriptors with the *close-on-exec* flag cleared are
+inherited in the child process, file descriptors with the flag set are
closed in the child process.
The flag can be set in two syscalls (one to get current flags, a second
@@ -113,8 +112,8 @@
The *close-on-exec* flag has no effect on ``fork()``: all file
descriptors are inherited by the child process. The `Python issue #16500
"Add an atfork module" <http://bugs.python.org/issue16500>`_ proposes to
-add a new ``atfork`` module to execute code at fork. It may be used to
-close automatically file descriptors at fork.
+add a new ``atfork`` module to execute code at fork, it may be used to
+close automatically file descriptors.
Issues with Inheritable File Descriptors
@@ -124,15 +123,14 @@
processes are not noticed, because they don't cause major bugs. It does
not mean that these bugs must not be fixed.
-Two example of common issues with inherited file descriptors:
+Two examples of common issues with inherited file descriptors:
-* On Windows, a directory cannot be removed until all file handles open
- in the directory are closed. It may explain why a temporary directory
- cannot be removed. The same issue can be seen with files, except if
- the file is temporary and was created with the ``FILE_SHARE_DELETE``
- flag (``O_TEMPORARY`` mode for ``open()``).
+* On Windows, a directory cannot be removed before all file handles open
+ in the directory are closed. The same issue can be seen with files,
+ except if the file was created with the ``FILE_SHARE_DELETE`` flag
+ (``O_TEMPORARY`` mode for ``open()``).
* If a listening socket is leaked in a child process, the socket address
- cannot be reused until the parent and child processes terminated. For
+ cannot be reused before the parent and child processes terminated. For
example, if a web server spawn a new program to handle a process, and
the server restarts while the program is not done: the server cannot
start because the TCP port is still in use.
@@ -146,9 +144,9 @@
An untrusted child process can read sensitive data like passwords and
take control of the parent process though leaked file descriptors. It is
-for example a known vulnerability to escape from a chroot. With a leaked
-listening socket, a child process can accept new connections to read
-sensitive data.
+for example a way to escape from a chroot. With a leaked listening
+socket, a child process can accept new connections to read sensitive
+data.
Atomic Creation of non-inheritable File Descriptors
@@ -156,22 +154,19 @@
In a multithreaded application, a inheritable file descriptor can be
created just before a new program is spawn, before the file descriptor
-is made non-inheritable. In this case, fhe file descriptor is leaked to
+is made non-inheritable. In this case, the file descriptor is leaked to
the child process. This race condition could be avoided if the file
descriptor is created directly non-inheritable.
FreeBSD, Linux, Mac OS X, Windows and many other operating systems
support creating non-inheritable file descriptors with the inheritable
-flag cleared atomically at the creating of the file descriptor.
+flag cleared atomically at the creation of the file descriptor.
-On Windows, since at least Windows XP, the `SECURITY_ATTRIBUTES
-<http://msdn.microsoft.com/en-us/library/windows/desktop/aa379560%28v=vs.85%29.aspx>`_
-structure can be used to clear the ``HANDLE_FLAG_INHERIT`` flag: set
-*bInheritHandle* field to ``FALSE``. This structure cannot be used with
-sockets: a new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag was added in Windows
-7 SP1 and Windows Server 2008 R2 SP1 for ``WSASocket()``. If this flag
-is used on an older Windows verison (ex: Windows XP SP3),
-``WSASocket()`` fails with ``WSAEPROTOTYPE``.
+A new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag for ``WSASocket()`` was added
+in Windows 7 SP1 and Windows Server 2008 R2 SP1 to create
+non-inheritable sockets. If this flag is used on an older Windows
+version (ex: Windows XP SP3), ``WSASocket()`` fails with
+``WSAEPROTOTYPE``.
On UNIX, new flags were added for files and sockets:
@@ -221,14 +216,14 @@
Legend:
* "Atomic File": first version of the operating system supporting
- creating atomatically a non-inheritable file descriptor using
+ creating atomically a non-inheritable file descriptor using
``open()``
* "Atomic Socket": first version of the operating system supporting
- creating atomatically a non-inheritable socket
+ creating atomically a non-inheritable socket
* "X": not supported yet
-Status in Python 3.3
+Status of Python 3.3
--------------------
Python 3.3 creates inheritable file descriptors on all platforms, except
@@ -239,15 +234,15 @@
``os.O_CLOEXEC``, ``os.pipe2()`` and ``socket.SOCK_CLOEXEC``.
On UNIX, the ``subprocess`` module closes all file descriptors in the
-child process, except standard streams (0, 1, 2) and file descriptors of
-the *pass_fds* parameter. If the *close_fds* parameter is set to
-``False``, all inheritable file descriptors are inherited in the child
-process.
+child process by default, except standard streams (0, 1, 2) and file
+descriptors of the *pass_fds* parameter. If the *close_fds* parameter is
+set to ``False``, all inheritable file descriptors are inherited in the
+child process.
On Windows, the ``subprocess`` closes all handles and file descriptors
in the child process by default. If at least one standard stream (stdin,
stdout or stderr) is replaced (ex: redirected into a pipe), all
-inheritable handles are inherited in the child process
+inheritable handles are inherited in the child process.
All inheritable file descriptors are inherited by the child process
using the functions of the ``os.execv*()`` and ``os.spawn*()`` families.
@@ -288,7 +283,7 @@
--------------------------------
The following functions are modified to make newly created file
-descriptors as non-inheritable by default:
+descriptors non-inheritable by default:
* ``asyncore.dispatcher.create_socket()``
* ``io.FileIO``
@@ -306,21 +301,25 @@
* ``socket.socket()``
* ``socket.socket.accept()``
* ``socket.socket.dup()``
- * ``socket.socket.fromfd``
+ * ``socket.socket.fromfd()``
* ``socket.socketpair()``
+When available, atomic flags are used to make file descriptors
+non-inheritable. The atomicity is not guaranteed because a fallback is
+required when atomic flags are not available.
+
New Functions
-------------
* ``os.get_inheritable(fd: int)``: return ``True`` if the file
descriptor can be inherited by child processes, ``False`` otherwise.
-* ``os.set_inheritable(fd: int, inheritable: bool)``: set the
+* ``os.set_inheritable(fd: int, inheritable: bool)``: clear or set the
inheritable flag of the specified file descriptor.
These new functions are available on all platforms.
-On Windows, these functions accept also "file descriptors" of sockets:
+On Windows, these functions accept also file descriptors of sockets:
the result of ``sockobj.fileno()``.
@@ -329,7 +328,7 @@
* On UNIX, subprocess makes file descriptors of the *pass_fds* parameter
inheritable. The file descriptor is made inheritable in the child
- process after the ``fork()`` and before ``execv()``, the inheritable
+ process after the ``fork()`` and before ``execv()``, so the inheritable
flag of file descriptors is unchanged in the parent process.
* ``os.dup2(fd, fd2)`` makes *fd2* inheritable if *fd2* is ``0``
@@ -342,23 +341,24 @@
This PEP break applications relying on inheritance of file descriptors.
Developers are encouraged to reuse the high-level Python module
-``subprocess`` which handle the inheritance of file descriptors in a
+``subprocess`` which handles the inheritance of file descriptors in a
portable way.
Applications using the ``subprocess`` module with the *pass_fds*
parameter or using ``os.dup2()`` to redirect standard streams should not
be affected.
-Python does no more conform to POSIX, since file descriptors are made
-non-inheritable by default. Python was not designed to conform to POSIX,
-Python is designed to develop portable applications.
+Python does no more conform to POSIX, since file descriptors are now
+made non-inheritable by default. Python was not designed to conform to
+POSIX, but was designed to develop portable applications.
-Previous Work
-=============
+Related Work
+============
The programming languages Go, Perl and Ruby make newly created file
-descriptors non-inheritable: since Go 1.0, Perl 1.0 and Ruby 2.0.
+descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0
+(1987) and Ruby 2.0 (2013).
The SCons project overrides builtin functions ``file()`` and ``open()``
to make files non-inheritable on Windows:
@@ -382,18 +382,17 @@
Functions handling file descriptors should not handle standard streams
(file descriptors ``0``, ``1``, ``2``) differently.
-This option does not work on Windows. On Windows,
-``os.set_inheritable(fd, inheritable)`` (calling
-``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag)
-on file descriptor ``0`` (stdin), ``1`` (stdout) or ``2`` (stderr) fails
-with ``OSError(87, 'invalid argument')``. If ``os.dup2(fd, fd2)`` would
-always make *fd2* non-inheritable, the function would raise an exception
-when used to redirect standard streams.
+This option does not work on Windows. On Windows, calling
+``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag
+on standard streams (0, 1, 2) fails with the Windows error 87 (invalid
+argument). If ``os.dup2(fd, fd2)`` would always make *fd2*
+non-inheritable, the function would raise an exception when used to
+redirect standard streams.
Another option is to add a new *inheritable* parameter to ``os.dup2()``.
This PEP has a special-case for ``os.dup2()`` to not break backward
-compatibility on applications redirection standard streams before
+compatibility on applications redirecting standard streams before
calling the C function ``execv()``. Examples in the Python standard
library: ``CGIHTTPRequestHandler.run_cgi()`` and ``pty.fork()`` use
``os.dup2()`` to redict stdin, stdout and stderr.
--
Repository URL: http://hg.python.org/peps
More information about the Python-checkins
mailing list