Reading Live Output from a Subprocess

[Dubslow]

On Friday, April 6, 2012 3:37:10 AM UTC-5, Nobody wrote:

> In all probability, this is because the child process (pypy) is
> buffering its stdout, meaning that the data doesn't get passed to the OS
> until either the buffer is full or the process terminates. If it doesn't
> get passed to the OS, then the OS can't pass it on to whatever is on the
> read end of the pipe. In that situation, there is nothing that the parent
> process can do about it.
>
It's just a short test script written in python, so I have no idea how to even control the buffering (and even if I did, I still can't modify the subprocess I need to use in my script). What confuses me then is why Perl is able to get around this just fine without faking a terminal or similar stuff. (And also, this needs to work in Windows as well.) For the record, here's the test script:
######################################
#!/usr/bin/python

import time, sys
try:
	total = int(sys.argv[1])
except IndexError:
	total = 10

for i in range(total):
	print('This is iteration', i)
	time.sleep(1)

print('Done. Exiting!')
sys.exit(0)
######################################

> If it does, that would confirm the buffering hypothesis. Pexpect causes
> the child process' stdout to be associated with a pty.
> 
> The "stdout" stream created by the C library (libc) is initially
> line-buffered if it is associated with a tty (according to the isatty()
> function) and fully-buffered otherwise. The program can change the
> buffering with e.g. setvbuf(), or it can explicitly fflush(stdout) after
> each line, but this is up to the program, and is not something which can
> be controlled externally (short of "hacking" the child process with
> techniques such as ptrace() or LD_PRELOAD).
> 
> While the libc behaviour is occasionally inconvenient, it is mandated by
> the C standard (7.19.3p7):
> 
> 	As initially opened, the standard error stream is not fully
> 	buffered; the standard input and standard output streams are
> 	fully buffered if and only if the stream can be determined not
> 	to refer to an interactive device.
> 
> It's up to individual programs to force line-buffered output where
> appropriate (e.g. GNU grep has the --line-buffered switch, GNU sed has the
> -u switch, etc).

Well, they shouldn't assume they can determine what's interactive or not *grumble*. I take it then that setting Shell=True will not be fake enough for catching output live?

On Friday, April 6, 2012 1:43:35 PM UTC-5, shi wrote:
> Maybe this can help you?
> http://technogems.blogspot.com/2012/01/capture-colored-console-output-in.html

Maybe, but even if it does work as expected, it's a _lot_ of pain to go through, with separate threads, etc.. just to get something so blasted simple to work. It certainly isn't obvious. Thanks for the link though.