SNIP IT: Popen3 in a non-heinous wrapper

[Noah]

This is interesting. I'm working on something similar now. Your open
issue with the child app being required to flush early and often is
likely to remain an issue. I had the same problem when I first tried to
do this with pipes. I learned from the helpful people on this list and
from my own reading that applications  that use the Standard IO library
will likely get hung on a pipe.
	man -s 3 stdio

The Standard IO library has three states for a FILE *. These are:
_IOFBF for block buffered; _IOLBF for line buffered;  and _IONBF for
unbuffered. The STDIO lib will use block buffering when talking to a
block file descriptor such as a pipe. This is usually not helpful for
interactive programs.  The program may have put data in its output that
remains unflushed because the output buffer is not full; then the
program will go and deadlock while waiting for input -- because you
never send it any because you are still waiting for its output 
(still stuck in the STDIO's output buffer).

The "answer" is to use a pseudo-tty. A TTY device will force *line*
buffering (as opposed to block buffering). Line buffering means that
you will get  each line when the child program sends a line feed.  This
corresponds to the way most interactive programs operate --  send a
line of output then wait for a line of input.

I put "answer" in quotes because it's ugly solution and because there
is no POSIX standard for pseudo-TTY devices (even though they have a
TTY standard...). What would make more sense to me would be to have
some way to set a mode on a file descriptor so that it will tell the
STDIO to be line-buffered. I have investigated, and I don't think there
is a way to set the buffered state of a child process. The STDIO
Library does not maintain any external state in the kernel or whatnot,
so I don't think there is any way for you to alter it. I'm not quite
sure how this line-buffered/block-buffered state change happens
internally in the STDIO library. I think the STDIO lib looks at the
file descriptor and decides to change  behavior based on whether it's a
TTY or a block file (see isatty()).

I hope that this qualifies as helpful.

Yours,
Noah

-----Original Message-----
From: python-list-admin at python.org
[mailto:python-list-admin at python.org]On Behalf Of Phlip
Sent: Thursday, May 02, 2002 9:34 AM
To: python-list at python.org
Subject: SNIP IT: Popen3 in a non-heinous wrapper


Not Hyp:

Despite the copious and well-consolidated documentation on the
subject, I have assembled a single, coherent wrapper for Popen3 that
hides the cluster of functions one must call to beat use out of it.
Find it below my sig.

Remember that the app it calls must call "flush" early and often; this
is an open issue.

Reviews & upgrades welcome.

--
  Phlip
               http://www.c2.com/cgi/wiki?PhlIp
  --  Argue for your <limits.h> and
      sure enough, they'r yours        --


import popen2
from fcntl import fcntl, F_SETFL
from select import select


def Piper(command, outputSink, errorSink):
	proc = popen2.Popen3(command, capturestderr = 1)
	O_NONBLOCK = 04000
	fcntl(proc.tochild.fileno(), F_SETFL, O_NONBLOCK)
	fcntl(proc.fromchild.fileno(), F_SETFL, O_NONBLOCK)
	fcntl(proc.childerr.fileno(), F_SETFL, O_NONBLOCK)

	while 1:
		w = []
		selectables = [proc.childerr, proc.fromchild]
		r, w, e = select(selectables, w, selectables)

		for i in r + e:
			if i == proc.childerr:
				err = proc.childerr.read()
				if err:  errorSink(err)

			if i == proc.fromchild:
				all = proc.fromchild.read()
				if all:  outputSink(all)

		got = proc.poll()
		if got != -1:
			break

	return got
			

if __name__ == '__main__':

	def outputSink(output):
		print output

	def errorSink(error):
		 print '***********************', error

	Piper(
		'ls . ; ls non_existent_folder ; ls .',
		outputSink,
		errorSink,
		)