Problems with background processes on Windows

[geoffbache]

On Mar 30, 6:57 am, Gabriel Genellina <gagsl-... at yahoo.com.ar> wrote:
> Gabriel Genellina <gagsl-py2 <at> yahoo.com.ar> writes:
>
>
>
> > En Sat, 28 Mar 2009 06:03:33 -0300, geoffbache <geoff.bache <at>
> jeppesen.com>  
> > escribió:
>
> > > Well yes, but the point is surely that the standard output of the
> > > background sleeping process is pointed to a different location? (you
> > > can replace the null device with a file name of your choice and the
> > > point is the same.) This process should not have any connection to the
> > > standard output of sleep.py, and hence we shouldn't need to wait for
> > > it to finish when collecting the standard output of sleep.py, surely?
> > > (Even explicitly calling sys.stdout.close() in sleep.py doesn't seem
> > > to help)
>
> > Thesis: When the subprocess module creates the child process, it inherits  
> > the stdin/stdout/stderr handles from its parent (even if its own  
> > stdin/stdout/stderr are redirected; they're different). Until the  
> > grandchild process finishes, the grandparent stdout.read() won't return,  
> > because the pipe isn't closed until the last handle to it is closed.
>
> I've confirmed the above description.
>
> --- begin p0.py ---
> import subprocess,os
>
> p1 = subprocess.Popen(["python", "p1.py"],
>       stdout=subprocess.PIPE,
>       stderr=open(os.devnull, "wt"),
>       stdin=open(os.devnull))
> print p1.communicate()
> --- end p0.py ---
>
> --- begin p1.py ---
> import subprocess,sys,os,msvcrt
>
> subprocess.Popen(
>     ["python", "p2.py", str(msvcrt.get_osfhandle(sys.stdout.fileno()))],
>     stdout=open(os.devnull, "wt"),
>     stderr=open(os.devnull, "wt"),
>     stdin=open(os.devnull, "rt"))
> print "exit p1.py"
> --- end p1.py ---
>
> --- begin p2.py ---
> import sys, win32api, time, os
>
> with open("p2.pid","wt") as f: f.write("%d" % os.getpid())
> win32api.CloseHandle(int(sys.argv[1]))
> time.sleep(30)
> --- end p2.py ---
>
> p2 has to close the inherited file handle corresponding to p1's stdout. Then,
> when p1 itself finishes, the writing end of the pipe is actually closed and p0
> can continue.
>
> C:\TEMP\subp>python p0.py
> ('exit p1.py\r\n', None)
>
> C:\TEMP\subp>type p2.pid
> 3018
> C:\TEMP\subp>tasklist | find "python.exe"
> python.exe                  3018                         0     4.304 KB
>
> I'm unsure this could be considered a bug in subprocess - the documentation
> says that parameter close_fds=True is not supported on Windows (so, the child
> process inherits all open files, and this includes the parent's
> stdin/stdout/stderr).
>
> At the end, this is due to the fact that file handles 0, 1, 2 have no special
> significance on Windows - it's the C runtime library which makes such things
> special, not the OS.

Thanks for this investigation.

I'm not the right person to pronounce on whether this is a Python bug
or not but I certainly found the behaviour surprising and unhelpful.
When you redirect output you expect it to be redirected, not for some
connection to be invisibly maintained to the original
location. I still don't really understand why sys.stdout.close() in p1
doesn't help either, is there some other way I can just ditch the
standard output from within p1?

Actually, I'm quite happy if somebody can tell me how to work around
it, this is just a test I wrote to simulate some behaviour in my
system (which is the thing calling communicate(), i.e. p0). So both
the child process and grandchild process are just part of the test
(the real versions aren't python programs and p2 runs remotely).

Is there anything I can do to either of these processes to fix this
issue without importing non-standard libraries (i.e. import win32api
is out)?

/Geoff