running a pipe of commands from python

[Donn Cave]

Quoth Harald Kirsch <kirschh at lionbioscience.com>:
[ ... re module "pipes" ...]

| This also uses /bin/sh. And all this although python has all the
| necessary ingredients like fork/exec/wait/signal to do the plumbing
| itself. Strange.
|
| I think I'll stick to my own hack, the doc of which currently reads
| like this:
|
| def execpipe(cmds):
|   """
|   runs the commands given in the list cmds in a pipe with their stdout
|   and stdin plumbed together. Every element of cmds must itself be a
|   list giving the name of the command as the first element and the
|   rest of the list being its arguments.
|
|   If any of the elements of cmds is a pair, then the first must be the
|   command to run and the 2nd must be a file descriptor for output
|   which is then connected to the commands stderr. By default, the
|   stderr is the same as that of the calling script.
|   """
|
| The implementation seems to work for most cases. Not all signals which
| cause the parent to die are distributed properly to the childs.

You might in some cases want to dup unit 2 ("stderr") from the
output pipe on unit 1.  For example, if an upstream process in
the pipeline is something like "make" that generates a session
output somewhat indiscriminately on units 1 and 2.  That would
appear kind of hard to specify by supplying the descriptor in
advance.

Python processes in the pipeline would be feasible and maybe useful.
I.e., fork but no exec.

Occasionally you may want more control over argv[0] vs. the file
name to be exec'd.  Since execve() provides this control, it's
kind of a shame to design it out.  Environment variables likewise.

Remember to close pipe file descriptors accidentally inherited from
parent processes, lest they hold the pipe open when its intended
writer has exited.  Remember to trap all exceptions inside each
child fork.  Think about where to go from "can't exec".

It may be useful to allow pipelines to be components of a pipeline.

Generally speaking, the possibilities are numerous enough that
even UNIX shells, built to do this for a living, support only
the most popular options, so don't feel obligated to implement
anything unless you have a use for it.  Your objection to the
shell based functions like system() is legitimate, they can
be a hazard if the command data isn't exactly known in advance.
(I wouldn't be very concerned about the overhead, though.)

A process class would obviously make the system more flexible.

Signals are normally delivered to a process, but the tty
driver delivers them to the foreground process group, so
one might not be too surprised to see different results
from a kill vs. ctrl-C.  There might be situations where
this would be a problem, but I can't think of any right
off hand.  All the processes in your pipeline will be in
the same process group as the parent.

	Donn Cave, donn at u.washington.edu