Reading from sys.stdin reads the whole file in

[Chris Angelico]

On Wed, Aug 27, 2014 at 3:19 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> When I pipe one to the other, I expect each line to be printed as they
> arrive, but instead they all queue up and happen at once:

You're seeing two different problems here. One is the flushing of
stdout in out.py, as Marko mentioned, but it's easily proven that
that's not the whole issue. Compare "python out.py" and "python
out.py|cat" - the latter will demonstrate whether or not it's getting
flushed properly (the former, where stdout is a tty, will always flush
correctly).

But even with that sorted, iterating over stdin has issues in Python
2. Here's a tweaked version of your files (note that I cut the sleeps
to 2 seconds, but the effect is the same):

rosuav at sikorsky:~$ cat out.py
import time

print("Hello...",flush=True)
time.sleep(2)
print("World!",flush=True)
time.sleep(2)
print("Goodbye!",flush=True)
rosuav at sikorsky:~$ cat slurp.py
from __future__ import print_function
import sys
import time

for line in sys.stdin:
        print(time.ctime(), line)
rosuav at sikorsky:~$ python3 out.py|python slurp.py
Wed Aug 27 16:00:16 2014 Hello...

Wed Aug 27 16:00:16 2014 World!

Wed Aug 27 16:00:16 2014 Goodbye!

rosuav at sikorsky:~$ python3 out.py|python3 slurp.py
Wed Aug 27 16:00:19 2014 Hello...

Wed Aug 27 16:00:21 2014 World!

Wed Aug 27 16:00:23 2014 Goodbye!

rosuav at sikorsky:~$


With a Py2 consumer, there's still buffering happening. With a Py3
consumer, it works correctly. How to control the Py2 buffering,
though, I don't know.

ChrisA