[Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

[Guido van Rossum]

On Tue, Jul 23, 2013 at 4:21 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 23/07/2013 11:45pm, Victor Stinner wrote:
>>
>> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
>> *handles*, as indicated in its name. On Windows, file*descriptors*
>> are never inherited (are always closed) in child processes. I don't
>> think that it is possible to inherit file descriptors on Windows.
>
>
> Actually, you can inherit fds if you use os.spawnv() instead of
> subprocess.Popen().

Wow. Indeed you can -- I just tested this myself. How is this
accomplished? I guess the CRT has a backchannel to talk to itself when
it creates a process using spawn*?

This is the only reason I can think of for the odd default in the CRT
of opening file descriptors inheritable by default, which Victor
discovered. (But it doesn't explain why os.pipe() creates
uninheritable fds.)

If it weren't for this I would definitely vote to change the default
on Windows throughout the stdlib to create file descriptors whose
handles aren't inheritable. (Perhaps with a different policy for
stdin/stdout/stderr, which seem to be treated specially at the handle
level.)

I'm about ready to give up hope that we'll ever have a decent way to
deal with this.

But I'm also ready to propose that all this is such a mess that we
*should* change the default fd/handle inheritance to False, *across
platforms*, and damn the torpedoes -- i.e. accept breaking all
existing 3rd party UNIX code for subprocess creation that bypasses the
subprocess module, as well as breaking uses of os.spawn*() on both
platforms that depend on FD inheritance beyond stdin/stdout/stderr).

With the new, sane default, all we need instead of PEP 446 is a way to
make an FD inheritable after it's been created, which can be a single
os.make_inheritable(fd) call that you must apply to the fileno() of
the stream or socket object you want inherited (or directly to a FD
you created otherwise, e.g. with os.pipe()). On Windows, this should
probably only work with os.spawn*(), since otherwise you need *handle*
inheritance, not *FD* inheritance, and that's a non-portable concept
anyway.

We can fix multiprocessing any anything else in the stdlib that this
breaks, I presume.

To reduce the need for 3rd party subprocess creation code, we should
have better daemon creation code in the stdlib -- I wrote some damn
robust code for this purpose in my previous job, but it never saw the
light of day.

-- 
--Guido van Rossum (python.org/~guido)