Dialog with a process via subprocess.Popen blocks forever

[Donn Cave]

In article <mailman.4695.1173143028.32031.python-list at python.org>,
 "Gabriel Genellina" <gagsl-py2 at yahoo.com.ar> wrote:

> En Fri, 02 Mar 2007 14:38:59 -0300, Donn Cave <donn at u.washington.edu>  
> escribió:
> 
> > In article <mailman.4595.1172792873.32031.python-list at python.org>,
> >  "Gabriel Genellina" <gagsl-py2 at yahoo.com.ar> wrote:
> >
> >> On http://docs.python.org/lib/popen2-flow-control.html there are some
> >> notes on possible flow control problems you may encounter.
> >
> > It's a nice summary of one problem, a deadlock due to full pipe
> > buffer when reading from two pipes.  The proposed simple solution
> > depends too much on the cooperation of the child process to be
> > very interesting, though.  The good news is that there is a real
> > solution and it isn't terribly complex, you just have to use select()
> > and UNIX file descriptor I/O.  The bad news is that while this is
> > a real problem, it isn't the one commonly encountered by first
> > time users of popen.
> 
> More bad news: you can't use select() with file handles on Windows.

Bad news about UNIX I/O on Microsoft Windows is not really news.
I am sure I have heard of some event handling function analogous
to select, but don't know if it's a practical solution here.

> >> If you have no control over the child process, it may be safer to use a
> >> different thread for reading its output.
> >
> > Right - `I used threads to solve my problem, and now I have two
> > problems.'  It can work for some variations on this problem, but
> > not the majority of them.
> 
> Any pointers on what kind of problems may happen, and usual solutions for  
> them?
> On Windows one could use asynchronous I/O, or I/O completion ports, but  
> neither of these are available directly from Python. So using a separate  
> thread for reading may be the only solution, and I can't see why is it so  
> bad. (Apart from buffering on the child process, which you can't control  
> anyway).

I wouldn't care to get into an extensive discussion of the general
merits and pitfalls of threads.  Other than that ... let's look at
the problem:

 - I am waiting for child process buffered output
 - I have no control over the child process

Therefore I spawn a thread to do this waiting, so the parent thread
can continue about its business.  But assuming that its business
eventually does involve this dialogue with the child process, it
seems that I have not resolved that problem at all, I've only added
to it.  I still have no way to get the output.

Now if you want to use threads because you're trying to use Microsoft
Windows as some sort of a half-assed UNIX, that's a different issue
and I wouldn't have any idea what's best.

   Donn Cave, donn at u.washington.edu