Passing File Descriptors To Subprocesses

[Lawrence D’Oliveiro]

A few years ago I wrote a tool <https://github.com/ldo/apitizer> to do comparisons between different versions of the Android API. Then one day, after a Python update (from the dates, it had to have been some version of 3.2), it stopped working. It took quite a while--over a year--until I figured out what was happening, and I could move my code from Python 2 back to Python 3 again.

The various subprocess functions <https://docs.python.org/3/library/subprocess.html> have arguments called “close_fds” and “pass_fds”, which specify which file descriptors are to be left open in the child process. Yet no matter what I set these to, it seemed I could not pass my pipes to a subprocess.

What the docs *don’t* tell you is that these arguments do not control what happens after the exec. The file descriptors that are kept open are only those which do not have the FD_CLOEXEC flags set in their fcntl settings.

Remember how the subprocess functions work:
* first, a fork(2) call is executed, then
* the child process does execve(2) (or some convenience variant of this) to actually execute the command or program that you specified.

The “close_fds” and “pass_fds” args only matter in the first step. You need to specify these, *and* have the right CLOEXEC settings on those file descriptors for the second step. Leave out either one, and your child process does not get the file descriptors.

Python 3.4 has added special calls <https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors> to manage this CLOEXEC setting. My code uses fcntl <https://docs.python.org/3/library/fcntl.html>.

The docs for the subprocess module need to make this requirement clear.