Problems with background processes on Windows

[Gabriel Genellina]

Gabriel Genellina <gagsl-py2 <at> yahoo.com.ar> writes:
> En Sat, 28 Mar 2009 06:03:33 -0300, geoffbache <geoff.bache <at> 
jeppesen.com>  
> escribió:
> >
> > Well yes, but the point is surely that the standard output of the
> > background sleeping process is pointed to a different location? (you
> > can replace the null device with a file name of your choice and the
> > point is the same.) This process should not have any connection to the
> > standard output of sleep.py, and hence we shouldn't need to wait for
> > it to finish when collecting the standard output of sleep.py, surely?
> > (Even explicitly calling sys.stdout.close() in sleep.py doesn't seem
> > to help)
> 
> Thesis: When the subprocess module creates the child process, it inherits  
> the stdin/stdout/stderr handles from its parent (even if its own  
> stdin/stdout/stderr are redirected; they're different). Until the  
> grandchild process finishes, the grandparent stdout.read() won't return,  
> because the pipe isn't closed until the last handle to it is closed.

I've confirmed the above description.

--- begin p0.py ---
import subprocess,os

p1 = subprocess.Popen(["python", "p1.py"], 
      stdout=subprocess.PIPE,
      stderr=open(os.devnull, "wt"),
      stdin=open(os.devnull))
print p1.communicate()
--- end p0.py ---

--- begin p1.py ---
import subprocess,sys,os,msvcrt

subprocess.Popen(
    ["python", "p2.py", str(msvcrt.get_osfhandle(sys.stdout.fileno()))], 
    stdout=open(os.devnull, "wt"),
    stderr=open(os.devnull, "wt"),
    stdin=open(os.devnull, "rt"))
print "exit p1.py"
--- end p1.py ---

--- begin p2.py ---
import sys, win32api, time, os

with open("p2.pid","wt") as f: f.write("%d" % os.getpid())
win32api.CloseHandle(int(sys.argv[1]))
time.sleep(30)
--- end p2.py ---

p2 has to close the inherited file handle corresponding to p1's stdout. Then, 
when p1 itself finishes, the writing end of the pipe is actually closed and p0 
can continue.

C:\TEMP\subp>python p0.py
('exit p1.py\r\n', None)

C:\TEMP\subp>type p2.pid
3018
C:\TEMP\subp>tasklist | find "python.exe"
python.exe                  3018                         0     4.304 KB

I'm unsure this could be considered a bug in subprocess - the documentation 
says that parameter close_fds=True is not supported on Windows (so, the child 
process inherits all open files, and this includes the parent's 
stdin/stdout/stderr). 

At the end, this is due to the fact that file handles 0, 1, 2 have no special 
significance on Windows - it's the C runtime library which makes such things 
special, not the OS.