[issue42738] subprocess: don't close all file descriptors by default (close_fds=False)

[Alexey Izbyshev]

Alexey Izbyshev <izbyshev at ispras.ru> added the comment:

> As a concrete example, we have a (non-Python) build system and task runner that orchestrates many tasks to run in parallel. Some of those tasks end up invoking Python scripts that use subprocess.run() to run other programs. Our task runner intentionally passes an inheritable file descriptor that is unique to each task as a form of a keep-alive token; if the child processes continue to pass inheritable file descriptors to their children, then we can determine whether all of the processes spawned from a task have terminated by checking whither the last open handle to that file descriptor has been closed. This is particularly important when a processes exits before its children, sometimes uncleanly due to being force killed by the system or by a user.

I don't see how such scheme could be usable in a general-purpose build system. Python is just one example of a program that closes file descriptors for child processes, but there might be arbitrary more. In general, there is no guarantee that a descriptor would be inherited in a process tree if it contains at least one process running code that you don't fully control. To properly control process trees, I'd suggest to consider prctl(PR_SET_CHILD_SUBREAPER) or PID namespaces on Linux and job objects on Windows (don't know about other OSes).

> Regarding security, PEP 446 already makes it so that any files opened from within a Python program are non-inheritable by default, which I agree is a good default. One can make the argument that it's not Python's job to enforce a security policy on file descriptors that a Python process has inherited from a parent process, since Python cannot distinguish from descriptors that were accidentally or intentionally inherited.

While I agree that such argument could be made, closing FDs for children also affects FDs opened by C extensions, which are not subject to PEP 446. And this is important not only for security: for example, it's easy to create a deadlock if one process is blocked on a read() from a pipe, which it expects to be closed at a proper moment, but the pipe write end is leaked to some unrelated long-running process which keeps it alive indefinitely without being aware of it.

And again, even if subprocess didn't close FDs by default (or somehow closed only non-inherited FDs), how could this be used in a wider picture? Any third-party library (even written in Python) could still decide to close e.g. some range of descriptors, so one wouldn't be able to rely on FD inheritance in a general setting, just as one can't rely on inheritance of environment variables (works most of the time, but no promises). Traditional Unix things like inheritance across fork()/exec() and no O_CLOEXEC by default are convenient for quick hacks, but my impression is that such defaults are considered to be design bugs in the modern setting by many.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42738>
_______________________________________