Getting stdout and stderr from popen

[Stefan Heimann]

Donn Cave <donn at u.washington.edu> wrote::

> Quoth Jonathan Gardner <jgardn at alumni.washington.edu>:
>| Stefan Heimann scribbled with his keyboard:
>|> I want to launch an external process and have stdout and stderr
>|> redirected to one output stream,
>|
>| You are on Unix, right?
>|
>|> something like
>|> 
>|> f = os.popen('cvs status 2>&1')
>|> 
>|> does the right thing, but does not work on all platforms.
> 
> Apparently he is not on UNIX?  That's the right answer, as long
> as popen() runs the UNIX shell.

I am on unix but I want it to work on windows, too.

> 
>|> If I write
>|> 
>|> pin, pout, perr = os.popen3('cvs status')
>|> 
>|> I get the output of stderr and stdout but not in the right order.
>|
>| What do you mean in the right order? Do you mean that it is returning in, 
>| err, out or that err and out are spitting out stuff simultaneously?

the latter one.

> 
> Probably not simultaneous, because the C I/O library probably has
> thread interlocks that prevent it, but if you want a single time
> ordered stream, you'd need time stamps to reassemble it out of
> two.  For example, suppose you record the output of "make", which
> will be both output and error streams.  If you read them separately,
> you won't be able to tell which compiler errors belong to which
> make commands.

That's what I need! Is it really so complicated? I think it should be
common that someone needs to read stdout and stderr ordered by time.

> 
>| Okay, let's talk about what is really happening deep underneath the hood.
>|
>| These popen functions are a shorthand for a very common thing in the world 
>| of Unix. It goes something like this:
>|
>| 1) make three pipes (in, out, err)
>| 2) fork.
> 
> I beg your pardon, but that's really not common at all.  Most UNIX
> pipes are created using the shell, which makes a pipe for output only.
> If you want a separate pipe for error, you're in for some gymnastics -
>   (cvs status 2>&3 | sed s/^/out:/) 3>&1 1>&2 | sed s/^/err:/
> And there's no Bourne shell formula at all to get all three on pipes.
> More recent shells have features that could be exploited for that,
> but it gets hairy.
> 
> This is not just nitpicking, because there are real good reasons why
> it isn't all that common.  It's surprisingly difficult to get two
> processes to cooperate reliably over multiple pipes.  That's my second
> point here - even if separate output and error streams are OK in terms
> of the data, they're distinctly more difficult to deal with if you
> don't make some simplifying assumptions about how the other process
> is going to behave.
> 
> The shell syntax to route both streams into the pipe is really the
> right way to go, if it works.  If it doesn't work, we'd have to know
> why not, before suggesting any answer.  (If it doesn't work on Windows,
> then someone else is going to have to answer it - and the subject line
> probably should have included "Windows" in the first place.)

Hm, I haven't tested it on Windows, but I *know* that the shell syntax
won't work. I didn't think that it is so complicated ;-)

Bye Stefan