[python-win32] Inheritable sockets on Windows and multiprocessing

[Preston Landers]

Thanks to Eryk Sun who pointed out that you can now just directly pass the
socket object to the child process in multiprocessing (instead of the
fileno int). I confirmed that worked for my use case.

http://bugs.python.org/issue28906

thanks,
Preston


On Tue, Dec 6, 2016 at 5:11 PM Preston Landers <planders at gmail.com> wrote:

Hello,

Please excuse me if this is not the appropriate forum. I'm porting a
Python 2.6 based application to Python 3.6. This app uses a customized
version of the "flup" package to do FastCGI services on Windows using
the multiprocessing package. This requires a few sockets to be
inherited across processes - the main FastCGI protocol socket plus a
control socket.

In Python 2.6, the `socket.from_fd` function was not available on
Windows. However I patched Python's socketmodule.c to provide that
function using DuplicateHandle. In Python 2.6's version of
multiprocessing it spawned a process with CreateProcess and
bInheritHandles=True. This worked beautifully for me.

Now I'm trying to get this working after moving from Python 2.6 to
3.6. Fortunately, the socket module now has a working `socket.from_fd`
on Windows so I no longer have to patch that. Unfortunately for me,
though, the multiprocessing module now calls CreateProcess with
bInheritHandles=False. This causes my scenario to fail.

Here's a short test script which illustrates this problem:
https://gist.github.com/Preston-Landers/712fee10fb557cf0b5592b57561a7c08

If you run with an unpatched multiprocessing, it will fail with an error
like:

OSError: [WinError 10038] An operation was attempted on something that
is not a socket

If you patch multiprocessing to set bInheritHandles=True this now
works. (Change is in popen_spawn_win32.py where it does
_winapi.CreateProcess.)

I'm sure there's a good reason for that change in multiprocessing,
whether for security or for unrelated/undesired file handles being
passed.
https://www.python.org/dev/peps/pep-0446/#inheritance-of-file-descriptors-on-windows

However it does break my scenario and I don't see a way to tell
multiprocessing to allow certain handles to be inherited. The docs for
multiprocessing say "In particular, unnecessary file descriptors and
handles from the parent process will not be inherited." It would be
nice to have a way to tell it that my sockets are "necessary." You
would think that calling socket.set_inheritable(True) would do it. In
fact you must do that, but you must also pass bInheritHandles=True to
CreateProcess for it to actually work.

I do realize I could be going about this completely wrong, though. But
right now it looks like my immediate options are:

a) Go ahead and patch my copy of popen_spawn_win32.py to allow
inherited handles.

b) Try to rewrite things to not use multiprocessing at all and
directly spawn my processes instead. That's not attractive because
multiprocessing otherwise does what I need to do.

Are there any other options I'm missing?

thanks,
Preston
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-win32/attachments/20161208/feb75c29/attachment.html>