Running a command line program and reading the result as it runs

[Rob Wolfe]

Ian Simcock <Ian.Simcock at Internode.on.net> writes:

> Greetings all.
>
> I'm using Python 2.7 under Windows and am trying to run a command line
> program and process the programs output as it is running. A number of
> web searches have indicated that the following code would work.
>
> import subprocess
>
> p = subprocess.Popen("D:\Python\Python27\Scripts\pip.exe list -o",
>                      stdout=subprocess.PIPE,
>                      stderr=subprocess.STDOUT,
>                      bufsize=1,
>                      universal_newlines=True,
>                      shell=False)
> for line in p.stdout:
>     print line
>
> When I use this code I can see that the Popen works, any code between
> the Popen and the for will run straight away, but as soon as it gets
> to the for and tries to read p.stdout the code blocks until the
> command line program completes, then all of the lines are returned.
>
> Does anyone know how to get the results of the program without it blocking?

When file object is used in a for loop it works like an iterator
and then it uses a hidden read-ahead buffer. 
It might cause this kind of blocking.
You can read more details here (description of method ``next``):
http://docs.python.org/lib/bltin-file-objects.html

So basically non-blocking loop might look like this:

while True:
    line = p.stdout.readline()
    if not line: break
    print line

HTH,
Rob