Use of os.popen3() under WinNT4

[David Bolen]

David Niergarth <jdnier at execpc.com> writes:

> >>> import os
> >>> command = 'gswin32c -help'
> >>> child_stdout, child_stdin, child_stderr = os.popen3(command, mode='b')
> 
> child_stdout ends up as a file descriptor opened in write mode, while
> child_stdin is a file descriptor opened in read mode; in and out seems
> backwards here. In the above command, Ghostscript prints a help message and
> quits. To get at the help message, you need to read from stdin.

The os.popen# interface result tuple is not in the same order as that
for the popen2 module.  I forget the exact reason (popen2 was first I
think, but importing the win32all pipe module into the Python core for
Windows was different, and I think considered more regular, and
probably some other stuff).  I think there was some discussion on the
python-dev list about it all.

The net of it is that the os.popen# functions result tuple is not the
same as the popen2 module functions tuple for stdin/stdout:

    os.popen#       child_stdin, child_stdout, child_stderr
    popen2.popen#   child_stdout, child_stdin, child_stderr

You can find comments to this effect (both availability and a warning
about the inversion of the result) in the reference in 6.1.2 for the
OS module and 6.8 for the popen2 module.  Just don't depend on the
docstrings for the popen2 module to work for the os module.

Things are consistent across platform in each module though.  If you
use the os.popen# functions, then they use the builtin under Windows,
or wrap the popen2 module under Unix, but in the latter case reverse
the result.  Conversely, if you use the popen2 module functions, then
they wrap the builtin os.popen# functions under Windows, but reverse
the result.

> More importantly, if you run a command that doesn't immediately terminate,
> trying to read from child_stdin or child_stderr before doing a
> child_stdout.close() causes a hang. (The only way I've found to recover is
> to kill the parent Python process.) Is there a way to tell if the process
> has terminated before trying a .read() or, better yet, is there a way to
> signal or force the process to finish?

Deadlock is always a risk when you are in control of both the stdin
and stdout handles for the child process, depending on the interaction
of how both you and the child process handle I/O.  First, let's invert
your handles to their reality, so the problem becomes hanging while
trying to read from child_stdout or child_stderr before doing a
child_stdin.close() - which sounds more sensible, or at least
explainable.

In such a case, what I expect is happening is you anticipate your read
eventually terminating (with an empty result) when the process
finishes and exits (closing its stdout).  But the process is still
waiting for further input on its stdin before it wants to exit, and
you're neither providing more input nor have you closed it to signal
to the process that you are done.  So you both wait for more "input".
Deadlock.

In this case, since you know that this is how this particular process
behaves, the simplest approach may just be designing your application
so that you know you're going to close the stdin of the child before
you ever expect to see the stdout close.  Since presumably you are
feeding input to the child, just close its stdin once you are done
with your input.

A more general mechanism is to handle the stdin/stdout flows in
separate threads.  That way, even if your thread reading the stdout is
blocked, presumably the thread that is supplying stdin to the child
will eventually finish and close that file, thus freeing up your
reading thread.  Note that even in this case there is a risk of
deadlock if you design badly so that you end up in a state where you
aren't draining the child's stdout, but your thread to drain the
stdout is coupled to the feeding of stdin, which may then block
because the child isn't reading it until it can get its stdout done.
But in a threaded model that's normally not too hard to avoid.

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/