??? Re: popen2 bug?

[Quinn Dunkan]

On Tue, 18 Sep 2001 20:17:00 +0400 (MSD), Roman Suzi <rnd at onego.ru> wrote:
>
>On Tue, 18 Sep 2001, Roman Suzi wrote:
>
>>I came across unidentified difficulty when doing the following:
>>
>>>>> d = (("a"*120) + "\n") * 300     # prepare data
>>>>> import popen2
>>>>> a, b = popen2.popen2("grep a")   # pipe to/from grep setup
>>>>> b.write(d)                       # trying to write... takes forever!
>
>The same happens even when using "cat" instead of "grep"
>and also it sometimes doesn't matter if \n is at the end or not.
>
>That is, popen2.* are nearly unusable or I am very mistaken
>about their use.

This is a unix FAQ.

There are a few common traps here.  The first is that stdio can figure out
whether it's talking to a pipe or a "real user" and in the case of a pipe
proceeds to buffer everything.  This means that many stdio-using programs
simply can't have a two-way conversation with a pipe because they'll save up
all their output instead of writing it.

The second problem is that since write() and read() block when a pipe is
stopped up (the other end isn't reading) or empty (the other end isn't
writing), it's very easy to get in a deadlock.  I suspect your grep is
deadlocking because you write to grep, and grep tries to write back to you.
Since you're busy writing to grep, you're not reading from grep, and hence
grep blocks on write().  Since grep is now no longer reading, you block on
write().  Wedge.

Usual "solutions" (hacks) for number one are using ptys to pretend that your
program is a real person.  Expect does this.  Python also has a pty module.

For number two, you can use select() or poll() and non-blocking IO, or you
can have reader/writer threads.  Some systems will tell you how many bytes can
be read if you fstat() a pipe, but I don't think that's portable.

I've successfully used popen2 to control interactive programs before, but
those programs typically don't use stdio.  So popen2 is usable, just not as
useful as you might think.