Non-blocking pipes during subprocess handling

[Donn Cave]

In article <22q5q25jn7dfti18rv1tshj7os5a0gm27r at 4ax.com>,
 Tom Plunket <tomas at fancy.org> wrote:

> I'm using subprocess to launch, well, sub-processes, but now I'm
> stumbling due to blocking I/O.
> 
> Is there a way for me to know that there's data on a pipe, and possibly
> how much data is there so I can get it?  Currently I'm doing this:
> 
> 	process = subprocess.Popen(
> 		args,
> 		bufsize=1,
> 		universal_newlines=True,
> 		stdout=subprocess.PIPE,
> 		stderr=subprocess.PIPE)
> 
> 	def ProcessOutput(instream, outstream):
> 		text = instream.readline()
> 		if len(text) > 0:
> 			print >>outstream, text,
> 			return True
> 		else:
> 			return False


I think it would be fair to say that your problem is
not due to blocking I/O, so much as buffered I/O.  Since
you don't appear to need to read one line at a time, you
can detect and read data from the file descriptor without
any buffering.  Don't mix with buffered I/O, as this will
throw the select off.  From memory - better check, since
it has been a while since I wrote anything real like this
(or for that matter much of anything in Python) --


import select
def ProcessOutput(instream, outstream):
    fdr = [instream.fileno()]
    (r, w, e) = select.select(fdr, [], [], 0.0)
    for fd in r:
        text = os.read(fd, 4096)
        outstream.write(text)

   Donn Cave, donn at u.washington.edu