Possible problem with popen2 module

[Donn Cave]

In article <e838aa6e.0404300614.57c13c8a at posting.google.com>,
 alloydflanagan at comcast.net (A. Lloyd Flanagan) wrote:
> OK, I've got a weird one I haven't been able to figure out yet. 
> Admittedly I haven't had time to dig into the library source, but this
> behavior certainly doesn't seem right.  Here's a test case:
> 
> """Test program to demonstrate problem with popen2 module."""
> import popen2
> 
> def main (argv):
>     mycmd = 'python2.3 -c "for i in range(100000):\n   print i"'
>     p = popen2.Popen3(mycmd)
>     print 'result code is %d' % p.wait()
>     for line in p.fromchild.xreadlines():
>         print line,
> 
> if __name__ == '__main__':
>     import sys
>     main(sys.argv)
> 
> As you can see I'm using the popen2.Popen3 class to run a process
> which prints a lot of output.
> 
> Here's the problem: for small values of the constant in the range()
> call, e.g. 1000, this works as expected.  For larger values, e.g.
> 100,000, the program never returns from the p.wait() call after the
> child process completes.  It appears tbe waiting forever on the
> waitpid() call.

> I don't see this behavior with calling os.popen().  I DO see it with
> the current implementation of popen5() from the PEP.
> 
> Does anyone know why this is occurring?  Is this a bona-fide bug?  Or
> am I just being stupid somehow?

Well, I will leave it to you to decide how stupid this was.
Nice job on the problem report though, really covers all the
bases.

Pipes are `slow', fixed size devices.  You can write only some
small amount of data to a pipe, and then you have to wait for
the peer process on the other end to read that data and make
more room.  Waiting like (and waiting for the peer on reads)
is what makes them slow, which really means interruptible by
signals (just an aside.)

What would work the way you want is a disk file.  Redirect
output to a file, wait for the process to exit, and read the
file.  Pipes are for processes whose output you want to read
while in progress, and you must do that whether you want to
or not.

You don't have exactly this problem with popen() because you're
not really doing the same thing - it doesn't have a wait(),
just a close(), and close() closes the pipe first, which kills
the process so the wait works.

   Donn Cave, donn at u.washington.edu