Clueless: piping between 2 non-python processes

[Donn Cave]

Quoth Andrew Bennetts <andrew-pythonlist at puzzling.org>:
...
| Also, ideally you'd close all other file descriptors in the child process
| apart from these pipes, to avoid allowing the child processes to muck with
| files that the parent has open.

Sure, popen() does that.  For me, that's more to avoid keeping a file
descriptor open past its normal lifetime.  UNIX file descriptors are
an interesting parallel to Python's object system, and this is a lot
like a reference leak - you may expect some important finalization to
run on collection of the object, or just reclaim its memory, so you
don't want to see other objects get accidental references to it that
will keep it alive.  The resource is file descriptors instead of memory,
the finalization may be end of file seen by another process, deletion
of the file inode where no directory entry remains, etc.

| > 
| > |         os.execl(pathToA)
| > ... os.execl(pathToA, pathToA)
|
| Ah, good point.
|
| > ... Better enclose the whole child fork's Python code in try/finally,
| > ... with os._exit(113) in the finally block (where 113 is some distinctive
| > ... number.)  Otherwise exceptions will branch back out of this block into
| > ... code that you intended for the parent.
|
| Yeah, that's safer, although I can't think of a reason why an exception
| would be raised here (but better safe than sorry).

Eh, you mean because I fixed the one that would have been raised?
I often fail to think in advance of the reasons why my code will
raise an exception.

...
| Well, if these are the only child processes his program spawns, he can
| afford to just use os.wait, e.g. something like:
|
|     pid, status = os.wait()
|     if pid == pidA:
|         otherpid = pidB
|     elif pid == pidB:
|         otherpid = pidA
|     else:
|         assert 0, "This isn't supposed to happen"
|     
|     try:
|         os.kill(pidB, signal.SIGTERM)
|     except OSError:
|         # Already dead, it seems
|         pass
|     
|     # Make sure to reap both children
|     os.waitpid(otherpid, 0)  
|
| But polling using the os.WNOHANG flag would work too, although polling
| always feels less elegant to me.

Get over it.  Killing a process that's on its way to exiting normally
because you find polling distasteful, gives elegance a bad name.  The
point I guess I needed to make more explicitly is that along with the
poll, you need a delay to give B time to exit.

[... re deadlock potential in 2 way piping ]

| If buffering is a problem, the processes comminicating via pipes are welcome
| to call fflush() or change their I/O library's buffer settings as needed.
| This isn't significantly different to the problems you can encounter with
| TCP sockets, unless I'm misunderstanding you.

The reason it's a common problem is that it normally works through
the standard UNIX in/out/err system, which is predicated on disk files
or a simple producer/consumer pipe line.  That assumption is reflected
in how the applications behave, and it's relatively unusual to be able
to do anything about that.

	Donn Cave, donn at drizzle.com