Reading Live Output from a Subprocess

[Chris Rebert]

On Thu, Apr 5, 2012 at 11:57 PM,  <bunslow at gmail.com> wrote:
> Okay, I've been trying for days to figure this out, posting on forums, Googling, whatever. I have yet to find a solution that has worked for me. (I'm using Python 3.2.2, Ubuntu 11.04.) Everything I've tried has led to buffered output being spat back all at once after the subprocess terminates. I need this functionality because the script I am currently writing is only glue, and most of the work is done by a subprocess (which could potentially run for a long time).
<snip>
> Not a single one of the solutions above has worked for me. I've tried
>
> ######################################
> import subprocess as sub
> out = sub.Popen(["pypy", "5"], universal_newlines=True, stdout=sub.PIPE, stderr=sub.STDOUT, bufsize=1)
>
> line = out.stdout.readline()
> out.stdout.flush()
> while line:
>    print(line)
>    line = out.stdout.readline()
>    out.stdout.flush()

You're flush()-ing the wrong stream here. The process's stdout is an
incoming stream from Python's perspective, which makes flushing it
pointless AFAIK. You want to flush *Python*'s stdout (that you're
print()-ing to). Which would be:

import sys

for line in out.stdout:
    print(line)
    sys.stdout.flush()

> ######################################
>
> I've tried
>
> ######################################
> line = out.stdout.readline()
> while line:
>    print(line)
>    line = out.stdout.readline()

Less repetitively written:

while True:
    line = out.stdout.readline()
    if not line:
        break
    print(line)

Or more idiomatically written:

# I hope this was the first thing you tried?
for line in out.stdout:
    print(line)

> ######################################
>
> I've tried
>
> ######################################
> for line in out.readline():
>    print(line)

(I assume you meant "out.stdout"? "out" is a bad name anyway; it's a
process, not an output stream.)
This iterates over individual characters in the first line of the
output. In "for X in Y", Y only gets evaluated once.

> ######################################
>
> I've tried
>
> ######################################
> for line in out.communicate():
>    print(line)

That's the explicitly-documented-as-blocking method, the exact
opposite of what you want.

Cheers,
Chris