Reading Live Output from a Subprocess

[bunslow at gmail.com]

Okay, I've been trying for days to figure this out, posting on forums, Googling, whatever. I have yet to find a solution that has worked for me. (I'm using Python 3.2.2, Ubuntu 11.04.) Everything I've tried has led to buffered output being spat back all at once after the subprocess terminates. I need this functionality because the script I am currently writing is only glue, and most of the work is done by a subprocess (which could potentially run for a long time).

Here are some StackOverflow questions about this topic:
http://stackoverflow.com/questions/2525263/capture-subprocess-output
http://stackoverflow.com/questions/2996887/how-to-replicate-tee-behavior-in-python-when-using-subprocess
http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess
http://stackoverflow.com/questions/1183643/unbuffered-read-from-process-using-subprocess-in-python
http://stackoverflow.com/questions/527197/intercepting-stdout-of-a-subprocess-while-it-is-running

And a few others for good measure:
http://www.linuxquestions.org/questions/programming-9/python-how-do-you-capture-stdout-stream-from-external-program-612998/
http://devlishgenius.blogspot.com/2008/10/logging-in-real-time-in-python.html

Not a single one of the solutions above has worked for me. I've tried

######################################
import subprocess as sub
out = sub.Popen(["pypy", "5"], universal_newlines=True, stdout=sub.PIPE, stderr=sub.STDOUT, bufsize=1)

line = out.stdout.readline()
out.stdout.flush()
while line:
    print(line)
    line = out.stdout.readline()
    out.stdout.flush()
######################################

I've tried 

######################################
line = out.stdout.readline()
while line:
    print(line)
    line = out.stdout.readline()
######################################

I've tried

######################################
for line in out.readline():
    print(line)
######################################

I've tried

######################################
for line in out.communicate():
    print(line)
######################################

etc...

None have worked, and it seems that the while loops worked in Python 2.x (according to those links), but not in Python 3. (I am a two week old Python coder, and it's already tied for my strongest language, which is why I decided to start with Python 3.)

I've heard that the Pexpect module works wonders, but the problem is that relies on pty which is available in Unix only. Additionally, because I want this script to be usable by others, any solution should be in the standard library, which means I'd have to copy the Pexpect code into my script to use it.

Is there any such solution in the Python 3 Standard Library, and if not, how much of a thorn is this?

"There should be one-- and preferably only one --obvious way to do it."
Unfortunately, this is one case where the above is true for Perl but not Python. Such an example in Perl is 

open(PROG, "command |") or die "Couldn't start prog!";
	while (<PROG>) {
	         print "$_"; }

(Note that I do not know Perl and do not have any intentions to learn it; the above comes from the script I was previously copying and extending, but I imagine (due to its simplicity) that it's a common Perl idiom. Note however, that the above does fail if the program re-prints output to the same line, as many long-running C programs do. Preferably this would also be caught in a Python solution.)

If there is a general consensus that this is a problem for lots of people, I might consider writing a PEP.

Of course, my highest priority is solving the blasted problem, which is holding up my script at the moment. (I can work around this by redirecting the program to a tmp file and reading that, but that would be such a perilous and ugly kludge that I would like to avoid it if at all possible.)

Thanks,
Bill