Capturing stdout incrementally

[David Bolen]

Josiah Carlson <jcarlson at uci.edu> writes:

> Your experience in unix has colored your impression of popen on
> Windows. The trick with Windows is that pipes going to/from apps are
> not real file handles, nor do they support select calls (Windows
> select comes from Windows' underlying socket library).  If they did,
> then the Python 2.4 module Popen5 would not be required.

Pipes under Windows (at least for the built-in os.popen* calls) are
true OS file handles (in terms of Windows OS system handles), created
via a CreatePipe call which are connected to a child process created
with CreateProcess.  You are correct that you can't select on them,
but that's not because they aren't real file handles, but because
Winsock under Windows is the odd man out.  Sockets in Winsock aren't
equivalent to other native OS handles (they aren't the "real" file
handles), and select was only written to work with sockets.  That's
also why sockets can't directly play in all of the other Windows
synchronization mechanisms (such as WaitFor[Multiple]Object[s]) but
you have to tie a socket to a different OS handle first, and then use
that handle in the sychronization call.

> Popen5 is supposed to allow the combination of os.popen and os.system
> on all platforms.  You get pollable pipes and the signal that the
> program ended with.  As for how they did it on Windows, I believe they
> are using pywin32 or ctypes.

I'm certainly all for additional portability for child process
management - although the internal os.popen* calls under Windows
already give you the exit code of the child process (which does get
some unique values when the process terminates abruptly), just without
any simulated signal bits.

But implementing popen5 under Windows will still run into the same
problem (that of select simply not working for other OS system handles
other than sockets), so I agree that will be a challenge, since
presumably they'll want to return a handle that looks like a Python
file and thus does have to have the underlying OS handle for basic I/O
to work.

Last I saw about the module was in January with a PEP announcement on
python-dev, but the PEP still indicates no Windows support (the
example was built on top of the popen2 module), and the python-dev
discussion led to proposing a start with a pure Python module.  I
can't find any code in the current CVS tree related to popen5, so I'm
not sure of the status.

Of course, none of this changes the original question in this thread
in that if the child process is going to select output buffering based
on the "tty" or "console" aspect of the pipe to which its output is
connected, you can't override that from the calling program, but have
to deal with the program being executed (or more properly fake it out
so that the controlling pipe appears more tty/console like).  I doubt
any popen* changes will affect that.

-- David