Chaining programs with pipe

[Grant Edwards]

On 2007-08-22, Steve Holden <steve at holdenweb.com> wrote:
> Grant Edwards wrote:
>> On 2007-08-21, avishay <avishorp at yahoo.com> wrote:
>> 
>>> I'm trying to chain two programs with a pipe (the output of
>>> one feeding the input of the other). I managed to capture the
>>> output and feeding the input of each program independently
>>> with popen, but how do I tie them together?
>> 
>> On Unix, you do the same thing you would in C. Create a pipe
>> using os.pipe(), then run one program with stdout connected to
>> the "write" end of the pipe and the other program with stdin
>> connected to the "read" end of the pipe.
>> 
>> Or you can take a bit of a shortcut by letting the subprocess
>> module create the pipe for you:
>> 
>> http://docs.python.org/lib/node536.html
>> 
>>> Is there a solution that works equally on all platforms?
>> 
>> The doc page for subprocess doesn't say what platforms support
>> it.  I've had a lot of problems trying to use the subprocess
>> module on windows.  As is typical for Windows, there are all
>> sorts of special cases that either don't work at all or don't
>> work the way they should. You pays your money and you takes
>> your chances.
>> 
> Grant:
>
> I will shortly have to write some training material on using
> subprocess under Windows, so if you have any pointers to where
> your accumulated knowledge can be gleaned I would be grateful
> for the time saving.

There were two main problems I ran into:

 1) When you try to execute a program with with a pathname like
    \\host\path\to\prog.exe instead of R:\path\to\prog.exe,
    cmd.exe chokes and says it can't execute the file.  If you
    use a UNC path (the "\\" version) you have to use
    subprocess's shell=False option.  [I did find references to
    a patched version of cmd.exe on a few web pages, but didn't
    persue that option.]  os.system() has the same issue with
    UNC path names.

 2) In a wxPython app (possibly in non-console apps in
    general?), the child process's stdout and/or stderr didn't
    default to usable values. When the child tried to write to
    them you'd get crashes with rather cryptic error messages.
    Explicitly setting the child's stderr and stdout fixes
    that.  I don't remember os.system() having this problem,
    since it only cropped up after I switched to subprocess to
    try to solve 1).

In hindsight, these don't seem like big problems, but I tripped
over them at the same time and it took me _days_ to figure out
what was wrong.  I'm sure most Win32 programmers already knew
about those problems, but I'm a Unix guy who occasionally tries
to ship a Windows version of a Python app: the concept of a
process defaulting to having a stderr or stdout that wasn't
writable was utterly foreign to me.
    
-- 
Grant Edwards                   grante             Yow! Mr and Mrs PED, can I
                                  at               borrow 26.7% of the RAYON
                               visi.com            TEXTILE production of the
                                                   INDONESIAN archipelago?