How can I read streaming output of a subprocess

[Adam Skutt]

On May 2, 7:46 am, Kiuhnm <kiuhnm03.4t.yahoo.it> wrote:
> On 5/2/2012 13:08, Damjan Georgievski wrote:
>
> > I want to read the stream of an external process that I start with
> > Python. From what I've seen that's not possible with the subprocess module?
>
> Try with
>     cmd = 'your command here'
>     stdout = Popen(cmd, shell = True, stdout = PIPE, stderr = STDOUT).stdout

One should never, ever start a process with shell=True unless it's
absolutely necessary.   It is a waste of resources and an a security
issue waiting to happen.  Just say no.  Also, stderr shouldn't be
combined with stdout unless there is a need to mix the two streams,
which usually is not the case.

Also, one needs to hold on to the POpen object so the process can be
properly reaped later or use a with block.  This is where things get
tricky for most applications.  It gets a little extra tricky for 'ip
monitor'.

Assuming the intention is to write a command-line application, the
right thing to do _most likely_ is:
    with Popen(['ip', 'monitor', 'neigh'], stdout=PIPE) as proc:
        try:
           for line in proc.stdout:
               # Do your processing here, it need not be line-by-line.
        finally:
           process.terminate()

Calling process.terminate() explicitly is necessary since "ip monitor"
writes output forever.  This is not the right thing to do in all
situations, and the right thing to do depends on your application and
the command being executed as a subprocess.  It's not really possible
to generalize fully.

As such, your error-handling / cleanup code may end up being
completely different.    Don't use my code if it's not applicable to
your situation.  In any case, you need to cleanup after yourself,
which means waiting until the subprocess terminates.  It may also
include cleaning up the file objects for the pipes.  Using POpen
objects in a with block does both, so do that where you can.

The official Python documentation for the subprocess module is pretty
clear and worth reading.  I encourage you to look through it
carefully.  It does, unfortunately, gloss over which resources need to
be cleaned up and when.

Adam