Buffering problem using subprocess module

[Jp Calderone]

On 21 Jul 2005 06:14:25 -0700, "Dr. Who" <google at spiceaid.com> wrote:
>I am using the subprocess module in 2.4.  Here's the fragment:
>
>bufcaller.py:
>	import sys, subprocess
>	proc = subprocess.Popen('python bufcallee.py', bufsize=0, shell=True,
>stdout=subprocess.PIPE)
>	for line in proc.stdout:
>		sys.stdout.write(line)
>
>bufcallee.py:
>	import time
>	print 'START'
>	time.sleep(10)
>	print 'STOP'
>
>Although the documentation says that the output should be unbuffered
>(bufsize=0) the program (bufcaller) pauses for 10 seconds and then
>prints START immediately followed by 'STOP' rather than pausing 10
>seconds in between them.  Note that I made bufcallee a Python script
>for ease of the example but in the real-world problem I am trying to
>solve it is simply an executable.
>
>Any ideas?

There are a few places buffering can come into play.  The bufsize parameter to Popen() controls buffering on the reading side, but it has no effect on buffering on the writing side.  If you add a sys.stdout.flush() after the prints in the child process, you should see the bytes show up immediately.  Another possibility is to start Python in unbuffered mode (pass the -u flag, or set PYTHONUNBUFFERED in the environment), but obviously this only applies to Python programs.  Still another possibility (generally the nicest) is to use a PTY instead of a pipe: when the C library sees stdout is a pipe, it generally decides to put output into a different buffering mode than when it sees stdout is a pty.  I'm not sure how you use ptys with the subprocess module.

Hope this helps,

Jp