Clueless: piping between 2 non-python processes

[Donn Cave]

Quoth Andrew Bennetts <andrew-pythonlist at puzzling.org>:
| On Sat, Oct 25, 2003 at 08:16:33PM +0200, Michael Lehmeier wrote:
...
| > So the basic problem is:
| > - create two processes A and B
| > - pipe from A to B
| > - terminate A when B ends
|
| I think you can do something like (untested):

Well, this is really more than he asked for.  All we need here is 
the same thing as the shell does with 'a | b', but instead of forking
a from b as the shell would normally do it, both processes need to
be children of the Python program so that it can wait for either.
[More comments interleaved.]

|     import os, signal
|     
|     pathToA = '/usr/bin/A'
|     pathToB = '/usr/bin/B'
|     
|     # Create pipes
|     readA, writeB = os.pipe()
|     readB, writeA = os.pipe()

... We only need one pipe here, the second one.

|     # Create process A
|     pidA = os.fork()
|     if pidA == 0:   # child
|         os.dup2(readA.fileno(), 0)  # set read pipe to stdin
... omit the above line.
|         os.dup2(writeA.fileno(), 1) # set write pipe to stdout
... readA and writeA will be integer unit numbers, so omit ".fileno()"

|         os.execl(pathToA)
... os.execl(pathToA, pathToA)

... Better enclose the whole child fork's Python code in try/finally,
... with os._exit(113) in the finally block (where 113 is some distinctive
... number.)  Otherwise exceptions will branch back out of this block into
... code that you intended for the parent.

|     # Create process B
|     pidB = os.fork()
|     if pidB == 0:   # child
|         os.dup2(readB.fileno(), 0)  # set read pipe to stdin
|         os.dup2(writeB.fileno(), 1) # set write pipe to stdout
... omit above line, and see above for same comments.

|         os.execl(pathToB)
|
|     # Close file descriptors in the parent; it doesn't need them anymore
|     readA.close(); writeB.close(); readB.close(); writeA.close()
... os.close(readB), os.close(writeA)

|     # Wait for B to terminate
|     os.waitpid(pidB, 0)

OK, this is where the fun starts.  I understood the problem to be
that B may fail to exit when A exits, so I think we really want
to wait for A.  Or if the converse may also happen, then we need
to wait for either and then dispatch the other.  In any case I think
this is not too hard to figure out.  The most useful trick here is
the os.WNOHANG flag to waitpid, which will allow one or more waits
without blocking to see if B is really going to exit on its own.
You don't want to kill it unless you're fairly sure it's necessary,
assuming it's doing something useful enough to justify running it
in the first place.  kill should also be surrounded with try/except,
because on some platforms it may be an error to kill a process that
has exited even if it still hasn't been reaped.

|     # Kill A (even if it has finished already... this might need a try/except)
|     os.kill(pidA, signal.SIGTERM)
|
|     # Wait for A (to avoid zombies)
|     os.waitpid(pidA, 0)
|
| I've probably stuffed up some details, but I'm pretty sure that that's the
| basic idea.

Well, it's not much worse than the one that proposed a thread.

The bi-directional pipes you set up there can be a good thing, in
a case where that's what you need, but even then they're extremely
brittle, because pipes have a fixed, limited buffer size and because
C library I/O (including Python's fileobject) employs process internal
block buffering when writing to pipes.  The former means the pipe can
fill up when the reading process is dilatory, the latter means the
pipe may be empty at a point where the writing process has logically
written to it.  Often enough you find both conditions together.

	Donn Cave, donn at drizzle.com