skip to navigation
skip to content

Python Wiki

Python Insider Blog

Python 2 or 3?

Help Fund Python

[Python resources in languages other than English]

Non-English Resources

Add an event to this calendar.

Add an event to this calendar.

PEP:446
Title:Add new parameters to configure the inheritance of files and for non-blocking sockets
Version:fa873d5aed27
Last-Modified:2013-07-17 13:15:18 +0200 (Wed, 17 Jul 2013)
Author:Victor Stinner <victor.stinner at gmail.com>
Status:Draft
Type:Standards Track
Content-Type:text/x-rst
Created:3-July-2013
Python-Version:3.4

Abstract

This PEP proposes new portable parameters and functions to configure the inheritance of file descriptors and the non-blocking flag of sockets.

Rationale

Inheritance of file descriptors

The inheritance of file descriptors in child processes can be configured on each file descriptor using a close-on-exec flag. By default, the close-on-exec flag is not set.

On Windows, the close-on-exec flag is the inverse of HANDLE_FLAG_INHERIT. File descriptors are not inherited if the bInheritHandles parameter of the CreateProcess() function is FALSE, even if the HANDLE_FLAG_INHERIT flag is set. If bInheritHandles is TRUE, only file descriptors with HANDLE_FLAG_INHERIT flag set are inherited, others are not.

On UNIX, the close-on-exec flag is O_CLOEXEC. File descriptors with the O_CLOEXEC flag set are closed at the execution of a new program (ex: when calling execv()).

The O_CLOEXEC flag has no effect on fork(), all file descriptors are inherited by the child process. Futhermore, most properties file descriptors are shared between the parent and the child processes, except file attributes which are duplicated (O_CLOEXEC is the only file attribute). Setting O_CLOEXEC flag of a file descriptor in the child process does not change the O_CLOEXEC flag of the file descriptor in the parent process.

Issues of the inheritance of file descriptors

Inheritance of file descriptors causes issues. For example, closing a file descriptor in the parent process does not release the resource (file, socket, ...), because the file descriptor is still open in the child process.

Leaking file descriptors is also a major security vulnerability. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a known vulnerability to escape from a chroot.

Non-blocking sockets

To handle multiple network clients in a single thread, a multiplexing function like select() can be used. For best performances, sockets must be configured as non-blocking. Operations like send() and recv() return an EAGAIN or EWOULDBLOCK error if the operation would block.

By default, newly created sockets are blocking. Setting the non-blocking mode requires additional system calls.

On UNIX, the blocking flag is O_NONBLOCK: a pipe and a socket are non-blocking if the O_NONBLOCK flag is set.

Setting flags at the creation of the file descriptor

Windows and recent versions of other operating systems like Linux support setting the close-on-exec flag directly at the creation of file descriptors, and close-on-exec and blocking flags at the creation of sockets.

Setting these flags at the creation is atomic and avoids additional system calls.

Proposal

New cloexec And blocking Parameters

Add a new optional cloexec on functions creating file descriptors:

  • io.FileIO
  • io.open()
  • open()
  • os.dup()
  • os.dup2()
  • os.fdopen()
  • os.open()
  • os.openpty()
  • os.pipe()
  • select.devpoll()
  • select.epoll()
  • select.kqueue()

Add new optional cloexec and blocking parameters to functions creating sockets:

  • asyncore.dispatcher.create_socket()
  • socket.socket()
  • socket.socket.accept()
  • socket.socket.dup()
  • socket.socket.fromfd
  • socket.socketpair()

The default value of cloexec is False and the default value of blocking is True.

The atomicity is not guaranteed. If the platform does not support setting close-on-exec and blocking flags at the creation of the file descriptor or socket, the flags are set using additional system calls.

New Functions

Add new functions the get and set the close-on-exec flag of a file descriptor, available on all platforms:

  • os.get_cloexec(fd:int) -> bool
  • os.set_cloexec(fd:int, cloexec: bool)

Add new functions the get and set the blocking flag of a file descriptor, only available on UNIX:

  • os.get_blocking(fd:int) -> bool
  • os.set_blocking(fd:int, blocking: bool)

Other Changes

The subprocess.Popen class must clear the close-on-exec flag of file descriptors of the pass_fds parameter. The flag is cleared in the child process before executing the program; the change does not change the flag in the parent process.

The close-on-exec flag must also be set on private file descriptors and sockets in the Python standard library. For example, on UNIX, os.urandom() opens /dev/urandom to read some random bytes and the file descriptor is closed at function exit. The file descriptor is not expected to be inherited by child processes.

Rejected Alternatives

PEP 433

The PEP 433 entitled "Easier suppression of file descriptor inheritance" is a previous attempt proposing various other alternatives, but no consensus could be reached.

This PEP has a well defined behaviour (the default value of the new cloexec parameter is not configurable), is more conservative (no backward compatibility issue), and is much simpler.

Add blocking parameter for file descriptors and use Windows overlapped I/O

Windows supports non-blocking operations on files using an extension of the Windows API called "Overlapped I/O". Using this extension requires to modify the Python standard library and applications to pass a OVERLAPPED structure and an event loop to wait for the completion of operations.

This PEP only tries to expose portable flags on file descriptors and sockets. Supporting overlapped I/O requires an abstraction providing a high-level and portable API for asynchronous operations on files and sockets. Overlapped I/O are out of the scope of this PEP.

UNIX supports non-blocking files, moreover recent versions of operating systems support setting the non-blocking flag at the creation of a file descriptor. It would be possible to add a new optional blocking parameter to Python functions creating file descriptors. On Windows, creating a file descriptor with blocking=False would raise a NotImplementedError. This behaviour is not acceptable for the os module which is designed as a thin wrapper on the C functions of the operating system. If a platform does not support a function, the function should not be available on the platform. For example, the os.fork() function is not available on Windows.

UNIX has more flag on file descriptors: O_DSYNC, O_SYNC, O_DIRECT, etc. Adding all these flags complicates the signature and the implementation of functions creating file descriptor like open(). Moreover, these flags do not work on any file type, and are not portable.

For all these reasons, this alternative was rejected. The PEP 3156 proposes an abstraction for asynchronous I/O supporting non-blocking files on Windows.