Piping processes works with 'shell = True' but not otherwise.

[Chris Rebert]

On May 24, 2013 7:06 AM, "Luca Cerone" <luca.cerone at gmail.com> wrote:
>
> Hi everybody,
> I am new to the group (and relatively new to Python)
> so I am sorry if this issues has been discussed (although searching for
topics in the group I couldn't find a solution to my problem).
>
> I am using Python 2.7.3 to analyse the output of two 3rd parties programs
that can be launched in a linux shell as:
>
>  program1 | program2
>
> To do this I have written a function that pipes program1 and program2
(using subprocess.Popen) and the stdout of the subprocess, and a function
that parses the output:
>
> A basic example:
>
> from subprocess import Popen, STDOUT, PIPE
> def run():
>   p1 = Popen(['program1'], stdout = PIPE, stderr = STDOUT)
>   p2 = Popen(['program2'], stdin = p1.stdout, stdout = PIPE, stderr =
STDOUT)

Could you provide the *actual* commands you're using, rather than the
generic "program1" and "program2" placeholders? It's *very* common for
people to get the tokenization of a command line wrong (see the Note box in
http://docs.python.org/2/library/subprocess.html#subprocess.Popen for some
relevant advice).

>   p1.stdout.close()
>   return p2.stdout
>
>
> def parse(out):
>   for row in out:
>     print row
>     #do something else with each line
>   out.close()
>   return parsed_output
>
>
> # main block here
>
> pout = run()
>
> parsed = parse(pout)
>
> #--- END OF PROGRAM ----#
>
> I want to parse the output of 'program1 | program2' line by line because
the output is very large.
>
> When running the code above, occasionally some error occurs (IOERROR:
[Errno 0]).

Could you provide the full & complete error message and exception traceback?

> However this error doesn't occur if I code the run() function as:
>
> def run():
>   p = Popen('program1 | program2', shell = True, stderr = STDOUT, stdout
= PIPE)
>   return p.stdout
>
> I really can't understand why the first version causes errors, while the
second one doesn't.
>
> Can you please help me understanding what's the difference between the
two cases?

One obvious difference between the 2 approaches is that the shell doesn't
redirect the stderr streams of the programs, whereas you /are/ redirecting
the stderrs to stdout in the non-shell version of your code. But this is
unlikely to be causing the error you're currently seeing.

You may also want to provide /dev/null as p1's stdin, out of an abundance
of caution.

Lastly, you may want to consider using a wrapper library such as
http://plumbum.readthedocs.org/en/latest/ , which makes it easier to do
pipelining and other such "fancy" things with subprocesses, while still
avoiding the many perils of the shell.

Cheers,
Chris
--
Be patient; it's Memorial Day weekend.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20130526/cfa81d07/attachment.html>