Running a command line program and reading the result as it runs

[Ian Simcock]

Peter Otten wrote:
> Ian Simcock wrote:
>
>> Greetings all.
>>
>> I'm using Python 2.7 under Windows and am trying to run a command line
>> program and process the programs output as it is running. A number of
>> web searches have indicated that the following code would work.
>>
>> import subprocess
>>
>> p = subprocess.Popen("D:\Python\Python27\Scripts\pip.exe list -o",
>>                        stdout=subprocess.PIPE,
>>                        stderr=subprocess.STDOUT,
>>                        bufsize=1,
>>                        universal_newlines=True,
>>                        shell=False)
>> for line in p.stdout:
>>       print line
>>
>> When I use this code I can see that the Popen works, any code between
>> the Popen and the for will run straight away, but as soon as it gets to
>> the for and tries to read p.stdout the code blocks until the command
>> line program completes, then all of the lines are returned.
>>
>> Does anyone know how to get the results of the program without it
>> blocking?
>
> The following works on my linux system:
>
> import subprocess
>
> p = subprocess.Popen(
>      ["ping", "google.com"],
>      stdout=subprocess.PIPE)
>
> instream = iter(p.stdout.readline, "")
>
> for line in instream:
>      print line.rstrip()
>
> I don't have Windows available to test, but if it works there, too, the
> problem is the internal buffer used by Python's implementation of file
> iteration rather than the OS.
>

Hmm... and so it comes full circle.

I thought that the inclusion of the iter call looked familiar so I 
checked my original code and found that it was there. I removed it when 
shrinking the code down to a minimal example for posting. Then, after 
removing it, which triggered the blocking, I changed the command to ping 
so that it's easier for anyone to test.

I've tried a copy of my original code using the ping command and it 
works fine.

So it looks like the python package manager pip must see that it's not 
going to a console and buffer the output, and my original code was not 
the problem.

So I can't do what I want, but it's interesting to know that for some 
reason the iter is required for the occasions when it can work.

Thanks to everyone who helped with this.

Ian Simcock