[Python-Dev] PEP 324: popen5 - New POSIX process module

[Peter Astrand]

On Sat, 3 Jan 2004, Martin v. Loewis wrote:

> >     - One "unified" module provides all functionality from previous
> >       functions.
> 
> I doubt this is a good thing. Different applications have different
> needs - having different API for them is reasonable.

I don't agree. I have used all of the existing mechanism in lots of apps,
and it's just a pain. There are lots of functions to choose between, but
none does what you really want. 


> >     - Cross-process exceptions: Exceptions happening in the child
> >       before the new process has started to execute are re-raised in
> >       the parent.  This means that it's easy to handle exec()
> >       failures, for example.  With popen2, for example, it's
> >       impossible to detect if the execution failed.
> 
> This is a bug in popen2, IMO. Fixing it is a good thing, but does not
> require a new module.

"Fixing popen2" would mean a break old applications; exceptions will 
happen, which apps are not prepared of. 


> >     - A hook for executing custom code between fork and exec.  This
> >       can be used for, for example, changing uid.
> 
> Such a hook could be merged as a keyword argument into the existing
> API.

Into which module/method/function? There is no one flexible enough. The 
case for redirecting only stderr is just one example; this is simple not 
possible with the current API. 


> >     - All combinations of file descriptor redirection is possible.
> >       For example, the "python-dialog" [2] needs to spawn a process
> >       and redirect stderr, but not stdout.  This is not possible with
> >       current functions, without using temporary files.
> 
> Sounds like a new function on the popen2 module.

To support all combinations, 12 different functions are necessary. Who 
will remember what popen2.popen11() means? 

 
> >     - Support for connecting several subprocesses (shell "pipe").
> 
> Isn't this available already, as the shell supports pipe creation,
> anyway?

With popen5, you can do it *without* using the shell. 


> >     - Universal newline support.
> 
> This should be merged into the existing code.

There's already a bug about this; bug 788035. This is what one of the 
comment says:

"But this whole popen{,2,3,4} section of posixmodule.c is so fiendishly
complicated with all the platform special cases that I'm loath to touch
it..."

I haven't checked if this is really true, though. 


> >     - A communicate() method, which makes it easy to send stdin data
> >       and read stdout and stderr data, without risking deadlocks.
> >       Most people are aware of the flow control issues involved with
> >       child process communication, but not all have the patience or
> >       skills to write a fully correct and deadlock-free select loop.
> 
> Isn't asyncore supposed to simplify that?

Probably not. The description says:

"This module provides the basic infrastructure for writing asynchronous
socket service clients and servers."

It's not obvious to me how this module could be use as a "shell backquote"  
replacement (which is what communicate() is about). It's probably possible
though; I haven't tried. Even if this is possible I guess we need some
kind of "entry" or "wrapper" method in the popen module to simplify things
for the user. My guess is that an communicate() method that uses asyncore
would be as long/complicated as the current implementation. The current 
implementation is only 68 lines, including comments. 


> So in short, I'm -1 on creating a new module, but +1 on merging
> most of these features into the existing code base - they are good
> features.

Well, I don't see how this could be done easily: The current API is not 
flexible enough, and some things (like cross-process exceptions) breaks 
compatibility. 

Writing a good popen module is hard. Providing cross-platform support (for 
Windows, for example) is even harder. Trying to retrofit a good popen 
implementation into an old API without breaking compatibility seems 
impossible to me. I'm not prepared to try. 


-- 
/Peter Åstrand <astrand at lysator.liu.se>