[Python-checkins] peps: PEP 433: cleanup, complete, reorganize
victor.stinner
python-checkins at python.org
Sun Jan 13 00:06:30 CET 2013
http://hg.python.org/peps/rev/7fd26f1f2ce1
changeset: 4667:7fd26f1f2ce1
user: Victor Stinner <victor.stinner at gmail.com>
date: Sun Jan 13 00:04:29 2013 +0100
summary:
PEP 433: cleanup, complete, reorganize
files:
pep-0433.txt | 322 +++++++++++++++++++++++---------------
1 files changed, 192 insertions(+), 130 deletions(-)
diff --git a/pep-0433.txt b/pep-0433.txt
--- a/pep-0433.txt
+++ b/pep-0433.txt
@@ -23,7 +23,9 @@
On UNIX, subprocess closes file descriptors greater than 2 by default since
Python 3.2 [#subprocess_close]_. All file descriptors created by the parent
-process are automatically closed.
+process are automatically closed. ``xmlrpc.server.SimpleXMLRPCServer`` sets
+the close-on-exec flag of the listening socket, the parent class
+``socketserver.BaseServer`` does not set this flag.
There are other cases creating a subprocess or executing a new program where
file descriptors are not closed: functions of the os.spawn*() family and third
@@ -31,6 +33,9 @@
descriptors are shared between the parent and the child processes which is
usually unexpected and causes various issues.
+This PEP proposes to continue the work started with the change in the
+subprocess, to fix the issue in any code, and not just code using subprocess.
+
Inherited file descriptors issues
---------------------------------
@@ -95,22 +100,10 @@
``fork()``. The close-on-exec flag only closes file descriptors after
``exec()``, and so after ``fork()`` + ``exec()``.
-Many functions of the Python standard library creating file descriptors are not
-changed by the PEP, and so will not have the close-on-exec flag set. Some
-examples:
-
- * ``os.urandom()``: on UNIX, it creates a file descriptor on UNIX to read
- ``/dev/urandom``. Adding an ``cloexec`` argument to ``os.urandom()`` does
- not make sense on Windows.
- * ``curses.windows.getwin()`` and ``curses.windows.putwin()`` creates a temporary file using ``fdopen(fd, "wb+");``
- * ``mmap.mmap()`` opens ``/dev/null`` using ``open("/dev/zero", O_RDWR);`` if
- ``MAP_ANONYMOUS`` is not defined.
- * If the ``PYTHONSTARTUP`` environment variable is set, the corresponding file
- is opened using ``fopen(startup, "r");``
- * ``python script.py`` opens ``script.py`` using ``fopen(filename, "r");``
- * etc.
-
-Third party modules creating file descriptors may not set close-on-exec flag.
+This PEP only change the close-on-exec flag of file descriptors created by the
+Python standard library, or by modules using the standard library. Third party
+modules not using the standard library should be modified to conform to this
+PEP. The new ``os.set_cloexec()`` function can be used for example.
Impacted functions:
@@ -129,9 +122,13 @@
XXX Should ``subprocess.Popen`` set the close-on-exec flag on file XXX
XXX descriptors of the constructor the ``pass_fds`` argument? XXX
+.. note::
+ See `Close file descriptors after fork`_ for a possible solution for
+ ``fork()`` without ``exec()``.
-Proposition
-===========
+
+Proposal
+========
This PEP proposes to add a new optional argument ``cloexec`` on functions
creating file descriptors in the Python standard library. If the argument is
@@ -160,41 +157,196 @@
The default value of the ``cloexec`` argument is ``False`` to keep the backward
compatibility.
+The close-on-exec flag will not be set on file descriptors 0 (stdin), 1
+(stdout) and 2 (stderr), because these files are expected to be inherited. It
+would still be possible to set close-on-exec flag explicitly using
+``os.set_cloexec()``.
+Drawbacks:
+ * Many functions of the Python standard library creating file descriptors are
+ cannot be changed by this proposal, because adding a ``cloexec`` optional
+ argument would be surprising and too many functions would need it. For
+ example, ``os.urandom()`` uses a temporary file on UNIX, but it calls a
+ function of Windows API on Windows. Adding a ``cloexec`` argument to
+ ``os.urandom()`` would not make sense. See `Always set close-on-exec flag`_
+ for an incomplete list of functions creating file descriptors.
+ * Checking if a module creates file descriptors is difficult. For example,
+ ``os.urandom()`` creates a file descriptor on UNIX to read ``/dev/urandom``
+ (and closes it at exit), whereas it is implemented using a function call on
+ Windows. It is not possible to control close-on-exec flag of the file
+ descriptor used by ``os.urandom()``, because ``os.urandom()`` API does not
+ allow it.
+Alternatives
+============
+
+Always set close-on-exec flag
+-----------------------------
+
+Always set close-on-exec flag on new file descriptors created by Python. This
+alternative just changes the default value of the new ``cloexec`` argument.
+
+If a file must be inherited by child processes, ``cloexec=False`` argument can
+be used.
+
+``subprocess.Popen`` constructor has an ``pass_fds`` argument to specify which
+file descriptors must be inherited. The close-on-exec flag of these file
+descriptors must be changed with ``os.set_cloexec()``.
+
+Example of functions creating file descriptors which will be modified to
+set close-on-exec flag:
+
+ * ``os.urandom()`` (on UNIX)
+ * ``curses.window.getwin()``, ``curses.window.putwin()``
+ * ``mmap.mmap()`` (if ``MAP_ANONYMOUS`` is not defined)
+ * ``oss.open()``
+ * ``Modules/main.c``: ``RunStartupFile()``
+ * ``Python/pythonrun.c``: ``PyRun_SimpleFileExFlags()``
+ * ``Modules/getpath.c``: ``search_for_exec_prefix()``
+ * ``Modules/zipimport.c``: ``read_directory()``
+ * ``Modules/_ssl.c``: ``load_dh_params()``
+ * ``PC/getpathp.c``: ``calculate_path()``
+ * ``Python/errors.c``: ``PyErr_ProgramText()``
+ * ``Python/import.c``: ``imp_load_dynamic()``
+ * TODO: ``PC/_msi.c``
+
+Many functions are impacted indirectly by this alternative. Examples:
+
+ * ``logging.FileHandler``
+
+Advantages of setting close-on-exec flag by default:
+
+ * There are far more programs that are bitten by FD inheritance upon
+ exec (see `Inherited file descriptors issues`_ and `Security`_) than
+ programs relying on it
+ (see `Applications using inherance of file descriptors`_).
+
+Drawbacks of setting close-on-exec flag by default:
+
+ * The os module is written as a thin wrapper to system calls (to functions of
+ the C standard library). If atomic flags to set close-on-exec flag are not
+ supported (see `Appendix: Operating system support`_), a single Python
+ function call may call 2 or 3 system calls (see `Performances`_ section).
+ * Extra system calls, if any, may slow down Python: see `Performances`_.
+ * It violates the principle of least surprise. Developers using the os module
+ may expect that Python respects the POSIX standard and so that close-on-exec
+ flag is not set by default.
+
+Backward compatibility: only a few programs rely on inherance of file
+descriptors, and they only pass a few file descriptors, usually just one.
+These programs will fail immediatly with ``EBADF`` error, and it will be simple
+to fix them: add ``cloexec=False`` argument or use
+``os.set_cloexec(fd, False)``.
+
+The ``subprocess`` module will be changed anyway to unset close-on-exec flag on
+file descriptors listed in the ``pass_fds`` argument of Popen constructor. So
+it possible that these programs will not need any fix if they use the
+``subprocess`` module.
+
+
+Add a function to set close-on-exec flag by default
+---------------------------------------------------
+
+An alternative is to add also a function to change globally the default
+behaviour. It would be possible to set close-on-exec flag for the whole
+application including all modules and the Python standard library. This
+alternative is based on the `Proposal`_ and adds extra changes.
+
+Add new functions:
+
+ * ``sys.getdefaultcloexec() -> bool``: get the default value of the
+ close-on-exec flag for new file descriptor
+ * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec
+ flag, the state of the flag can be overriden in each function creating a
+ file descriptor
+
+The major change is that the default value of the ``cloexec`` argument is
+``sys.getdefaultcloexec()``, instead of ``False``.
+
+When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by default,
+we have the same drawbacks than `Always set close-on-exec
+flag`_ alternative.
+
+There are additionnal drawbacks of having two behaviours depending on
+``sys.getdefaultcloexec()`` value:
+
+ * It is not more possible to know if the close-on-exec flag will be set or not
+ just by reading the source code.
+
+
+Close file descriptors after fork
+---------------------------------
+
+This PEP does not fix issues with applications using ``fork()`` without
+``exec()``. Python needs a generic process to register callbacks which
+would be called after a fork, see `Add an 'afterfork' module`_. Such
+registry could be used to close file descriptors just after a ``fork()``.
+
+Drawbacks:
+
+ * This alternative does not solve the problem for programs using ``exec()``
+ without ``fork()``.
+ * A third party module may call directly the C function ``fork()`` which will
+ not call "atfork" callbacks.
+ * All functions creating file descriptors must be changed to register a
+ callback and then unregister their callback when the file is closed. Or a
+ list of *all* open file descriptors must be maintained.
+ * The operating system is a better place than Python to close automatically
+ file descriptors. For example, it is not easy to avoid a race condition
+ between closing the file and unregistering the callback closing the file.
+
+
+open(): add "e" flag to mode
+----------------------------
+
+A new "e" mode would set close-on-exec flag (best-effort).
+
+This alternative only solves the problem for ``open()``. socket.socket() and
+os.pipe() do not have a ``mode`` argument for example.
+
+Since its version 2.7, the GNU libc supports ``"e"`` flag for ``fopen()``. It
+uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, F_SETFD, FD_CLOEXEC)``.
+With Visual Studio, fopen() accepts a "N" flag which uses ``O_NOINHERIT``.
+
Applications using inherance of file descriptors
================================================
+Most developers don't know that file descriptors are inherited by default. Most
+programs do not rely on inherance of file descriptors. For example,
+``subprocess.Popen`` was changed in Python 3.2 to close all file descriptors
+greater than 2 in the child process by default. No user complained about this
+behavior change.
+
Network servers using fork may want to pass the client socket to the child
-process. For example, a CGI server pass the socket client through file
-descriptors 0 (stdin) and 1 (stdout) using ``dup2()``.
+process. For example, on UNIX a CGI server pass the socket client through file
+descriptors 0 (stdin) and 1 (stdout) using ``dup2()``. This specific case is
+not impacted by this PEP because the close-on-exec flag is never set on file
+descriptors smaller than 3.
-Example of programs taking file descriptors from the parent process:
+To access a restricted resource like creating a socket listening on a TCP port
+lower than 1024 or reading a file containing sensitive data like passwords, a
+common practice is: start as the root user, create a file descriptor, create
+a child process, pass the file descriptor to the child process and exit.
+Security is very important in such use case: leaking another file descriptor
+would be a critical security vulnerability (see `Security`_). The root process
+may not exit but monitors the child process instead, and restarts a new child
+process and pass the same file descriptor if the previous child process
+crashed.
+Example of programs taking file descriptors from the parent process using a
+command line option:
+
+ * gpg: ``--status-fd <fd>``, ``--logger-fd <fd>``, etc.
+ * openssl: ``-pass fd:<fd>``
+ * qemu: ``-add-fd <fd>``
* valgrind: ``--log-fd=<fd>``, ``--input-fd=<fd>``, etc.
- * qemu: ``-add-fd <fd>`` command line option
- * GnuPG: ``--status-fd <fd>``, ``--logger-fd <fd>``, etc.
- * openssl command: ``-pass fd:<fd>``
* xterm: ``-S <fd>``
-On Linux, it is possible to use ``/dev/fd/<fd>`` filename to pass a file
-descriptor to a program expecting a filename. It can be used to pass a password
-for example.
-
-These applications only pass a few file descriptors, usually only one.
-Fixing these applications to unset close-on-exec flag should be easy.
-
-If the ``subprocess`` module is used, inherited file descriptors must be specified
-using the ``pass_fds`` argument (except if the ``close_fds`` argument is set
-explicitly to ``False``). So the ``subprocess`` module knows the list of file
-descriptors on which close-on-exec flag must be unset.
-
-File descriptors 0 (stdin), 1 (stdout) and 2 (stderr) are expected to be
-inherited and so should not have the close-on-exec flag. So a CGI server should
-not be impacted by this PEP.
+On Linux, it is possible to use ``"/dev/fd/<fd>"`` filename to pass a file
+descriptor to a program expecting a filename.
Performances
@@ -313,96 +465,6 @@
the close-on-exec flag is not set by default.
-Alternatives
-============
-
-Always set close-on-exec flag
------------------------------
-
-Always set close-on-exec flag on new file descriptors created by Python. This
-alternative just changes the default value of the new ``cloexec`` argument.
-
-``subprocess.Popen`` constructor has an ``pass_fds`` argument to specify which
-file descriptors must be inherited. The close-on-exec flag of these file
-descriptors must be changed with ``os.set_cloexec()``.
-
-If the close-on-exec flag must not be set, ``cloexec=False`` can be specified.
-
-Advantages of setting close-on-exec flag by default:
-
- * There are far more programs that are bitten by FD inheritance upon
- exec (see `Inherited file descriptors issues`_ and `Security`_) than
- programs relying on it.
- * Checking if a module creates file descriptors is difficult. For example,
- ``os.urandom()`` creates a file descriptor on UNIX to read ``/dev/urandom``
- (and closes it at exit), whereas it is implemented using a function call on
- Windows. It is not possible to control close-on-exec flag of the file
- descriptor used by ``os.urandom()``, because ``os.urandom()`` API does not
- allow it.
- * No need to add a new ``cloexec`` argument everywhere: functions creating
- file descriptors will read ``sys.getdefaultcloexec()`` to decide if the
- close-on-exec must be set or not. For example, adding an ``cloexec``
- argument to ``os.urandom()`` does not make sense on Windows.
-
-Drawbacks of setting close-on-exec flag by default:
-
- * The os module is written as a thin wrapper to system calls (to functions of
- the C standard library). If atomic flags are not supported, a single Python
- function call may now call 2 or 3 system calls (see `Performances`_
- section).
- * Extra system calls, if any, may slow down Python: see `Performances`_.
- * It violates the principle of least surprise. Developers using the os module
- may expect that Python respects the POSIX standard and so that close-on-exec
- flag is not set by default.
- * Only file descriptors created by the Python standard library will comply to
- ``sys.setdefaultcloexec()``. The close-on-exec flag is unchanged for file
- descriptors created by third party modules calling directly C functions.
- Third party modules will have to be modified to read
- ``sys.getdefaultcloexec()`` to make them comply to this PEP.
-
-
-Add a function to set close-on-exec flag by default
----------------------------------------------------
-
-An alternative is to add also a function to change globally the default
-behaviour. It would be possible to set close-on-exec flag for the whole
-application including all modules and the Python standard library. This
-alternative is based on the PEP but adds extra changes.
-
-Add new functions:
-
- * ``sys.getdefaultcloexec() -> bool``: get the default value of the
- close-on-exec flag for new file descriptor
- * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec
- flag, the state of the flag can be overriden in each function creating a
- file descriptor
-
-The major change is that the default value of the ``cloexec`` argument is
-``sys.getdefaultcloexec()``, instead of ``False``.
-
-When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by default,
-we have the same drawbacks than `Always set close-on-exec
-flag`_ alternative.
-
-There are additionnal drawbacks of having two behaviours depending on
-``sys.getdefaultcloexec()`` value:
-
- * It is not more possible to know if the close-on-exec flag will be set or not
- just by reading the source code.
-
-
-open(): add "e" flag to mode
-----------------------------
-
-A new "e" mode would set close-on-exec flag (best-effort).
-
-This API does not allow to disable explictly close-on-exec flag if it was
-enabled globally with ``sys.setdefaultcloexec()``.
-
-Note: Since its version 2.7, the GNU libc supports ``"e"`` flag for ``fopen()``.
-It uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, F_SETFD, FD_CLOEXEC)``.
-
-
Appendix: Operating system support
==================================
--
Repository URL: http://hg.python.org/peps
More information about the Python-checkins
mailing list