Running a command line program and reading the result as it runs

[Gertjan Klein]

Ian Simcock wrote:

> When I use this code I can see that the Popen works, any code between
> the Popen and the for will run straight away, but as soon as it gets to
> the for and tries to read p.stdout the code blocks until the command
> line program completes, then all of the lines are returned.
>
> Does anyone know how to get the results of the program without it blocking?

I have tried your code with "ping google.com" as command and got the 
same results; apparently something buffers the output. The result is 
different when using Python 3.3: there, the lines are printed as they 
come in. This seems to indicate a bug in the Python 2.7 implementation.

There are some bug reports on bugs.python.org that may be related; see 
for example:

http://bugs.python.org/issue15532

I have been playing around a bit with the suggested approach of using 
the io library directly. I managed to get unbuffered output, but 
unfortunately the program hangs when the subprocess is done. It can't 
even be terminated with Control-C, I have to use task manager to kill 
python.exe.

Below is as far as I got; perhaps someone with more experience with 
pipes knows how to fix this.

Regards,
Gertjan.


#!/usr/bin/env python2.7
# coding: CP1252

from __future__ import print_function
import subprocess
import io, os

def main():
     i, o = os.pipe()
     piperead = io.open(i, 'rb', buffering=1)

     p = subprocess.Popen(["ping", "google.com"],
                           stdout=o,
                           stderr=subprocess.PIPE,
                           bufsize=0,
                           shell=False)

     for line in piperead:
         print(line)

if __name__ == '__main__':
     main()