popen4 hanging on write

[Donn Cave]

In article 
<191262adf5b4ad1fb774c45d66fabebc at localhost.talkaboutprogramming.com>,
 "rinkworks" <sam at rinkworks.com> wrote:
> I'm having trouble with Popen4 locking up on me.  It appears to be
> unrelated to the actual command I'm running, because I made the command
> just "cat" (I'm on UNIX) to echo whatever input I sent it, and it's still
> locking.  The trick is, it works for small amounts of data.  After roughly
> 1.5K of input, it locks up and will not write any more data to the pipe. 
> I assume a buffer got full somewhere, and it's blocking the write call.
> 
> Meanwhile, I can't read the output, because if I do that, that blocks,
> too.
> 
> Anybody know how to resolve this problem?  Here is my current code:
> 
>         import popen2
>         pipe = popen2.Popen4("cat")
>         pipe.tochild.write(data)
>         pipe.tochild.close()
>         output = pipe.fromchild.read()
>         print output

You aren't the first person to encounter this problem,
but it's rare to see it laid out so neatly.

Try to put yourself in cat's place.  You're connected
to two pipes, one outbound and the other inbound, each
of them with some fixed capacity.  You also have your
own internal I/O buffers, for input and output.

Someone is writing data on the input pipe, so you read
that data and copy it to output.  Eventually, output
fills up, because no one is reading it.  The operating
system puts you to sleep waiting for space there.  At
this point, there may be some data left in your input
and output buffers, plus the contents of the output pipe.
Soon if not already, the input pipe is full too, and
the guy on the other end of it gets put to sleep.  Now
we're wedged.

Somehow, you must find a way to take a break from writing,
and read the data that's piling up in the output pipe.
That's a little tricky, because if it happens that there
isn't any data (or if you use fileobject read as above and
you request more data than there is), you may get wedged
again.  The select function may help, and in this case you
really need to use the posix I/O functions - os.read, os.write -
with the file descriptor from fromchild.fileno().

It gets complicated.  Popen4 is not an easy way to make
two processes talk to each other except in very limited
circumstances.  If you can use disk files, do.

   Donn Cave, donn at u.washington.edu