Running a command line program and reading the result as it runs

[Peter Otten]

Ian Simcock wrote:

> Greetings all.
> 
> I'm using Python 2.7 under Windows and am trying to run a command line
> program and process the programs output as it is running. A number of
> web searches have indicated that the following code would work.
> 
> import subprocess
> 
> p = subprocess.Popen("D:\Python\Python27\Scripts\pip.exe list -o",
>                       stdout=subprocess.PIPE,
>                       stderr=subprocess.STDOUT,
>                       bufsize=1,
>                       universal_newlines=True,
>                       shell=False)
> for line in p.stdout:
>      print line
> 
> When I use this code I can see that the Popen works, any code between
> the Popen and the for will run straight away, but as soon as it gets to
> the for and tries to read p.stdout the code blocks until the command
> line program completes, then all of the lines are returned.
> 
> Does anyone know how to get the results of the program without it
> blocking?

The following works on my linux system:

import subprocess

p = subprocess.Popen(
    ["ping", "google.com"],
    stdout=subprocess.PIPE)

instream = iter(p.stdout.readline, "")
        
for line in instream:
    print line.rstrip()

I don't have Windows available to test, but if it works there, too, the 
problem is the internal buffer used by Python's implementation of file 
iteration rather than the OS.