Getting stdout and stderr from popen

[Donn Cave]

Quoth Jonathan Gardner <jgardn at alumni.washington.edu>:
| Stefan Heimann scribbled with his keyboard:
|> I want to launch an external process and have stdout and stderr
|> redirected to one output stream,
|
| You are on Unix, right?
|
|> something like
|> 
|> f = os.popen('cvs status 2>&1')
|> 
|> does the right thing, but does not work on all platforms.

Apparently he is not on UNIX?  That's the right answer, as long
as popen() runs the UNIX shell.

|> If I write
|> 
|> pin, pout, perr = os.popen3('cvs status')
|> 
|> I get the output of stderr and stdout but not in the right order.
|
| What do you mean in the right order? Do you mean that it is returning in, 
| err, out or that err and out are spitting out stuff simultaneously?

Probably not simultaneous, because the C I/O library probably has
thread interlocks that prevent it, but if you want a single time
ordered stream, you'd need time stamps to reassemble it out of
two.  For example, suppose you record the output of "make", which
will be both output and error streams.  If you read them separately,
you won't be able to tell which compiler errors belong to which
make commands.

| Okay, let's talk about what is really happening deep underneath the hood.
|
| These popen functions are a shorthand for a very common thing in the world 
| of Unix. It goes something like this:
|
| 1) make three pipes (in, out, err)
| 2) fork.

I beg your pardon, but that's really not common at all.  Most UNIX
pipes are created using the shell, which makes a pipe for output only.
If you want a separate pipe for error, you're in for some gymnastics -
  (cvs status 2>&3 | sed s/^/out:/) 3>&1 1>&2 | sed s/^/err:/
And there's no Bourne shell formula at all to get all three on pipes.
More recent shells have features that could be exploited for that,
but it gets hairy.

This is not just nitpicking, because there are real good reasons why
it isn't all that common.  It's surprisingly difficult to get two
processes to cooperate reliably over multiple pipes.  That's my second
point here - even if separate output and error streams are OK in terms
of the data, they're distinctly more difficult to deal with if you
don't make some simplifying assumptions about how the other process
is going to behave.

The shell syntax to route both streams into the pipe is really the
right way to go, if it works.  If it doesn't work, we'd have to know
why not, before suggesting any answer.  (If it doesn't work on Windows,
then someone else is going to have to answer it - and the subject line
probably should have included "Windows" in the first place.)

	Donn Cave, donn at u.washington.edu