PEP 324: popen5 - New POSIX process module

[Peter Astrand]

> I've read the PEP, and I'd like to help with an implementation for
> Windows, if no-one else is doing it. 

Help is always appreciated. I haven't got very far yet. It might be useful 
to look at http://starship.python.net/~tmick/#process as well (but we 
should really try to get a smaller code base than this). 


>Initially, I'll probably use
> win32all (or maybe ctypes) but the intention is that longer term the
> necessary functions be migrated into a supporting C extension.

I like this approach. Ideally, popen5 should work on any Python 2.3+ 
system with win32all, or using the build-in support, when we have added 
that. So, when we are writing supporting C code, we should probably keep 
the interface from win32all. 


> 1. The preexec_* arguments have no meaning on Windows, where the
>    low-level functionality is CreateProcess (spawn) rather than
>    fork/exec. This isn't too crucial, as use of the argument can
>    either be ignored, or probably better, treated as an error on
>    Windows.

Yes, I've thought of this. I also think that these arguments (preexec_fn 
and preexec_arg) should be treated as errors. 


> 2. The method name poll() doesn't seem appropriate in a Windows
>    context (I don't know how it fits with Unix). Better might be
>    is_complete() returning True/False, and use returncode to get the
>    status separately.

Yes, perhaps. This needs some thought. 


> 3. The whole thing about the return code encoding both the exit status
>    and the signal number is very Unix-specific. Why not split them -
>    status and signal attributes, and maybe have is_complete() return 3
>    possible values - +1 = completed OK, 0 = still running, -1 = died
>    due to a signal.

Perhaps. poll() and wait() is mainly a heritage from popen2. I have no 
objections to change this, if we can come up with a good and clean 
solution. Your idea looks interesting, although I've not convinced on the 
name "is_complete()". 

Currently, popen5 provides both the unaltered "exit status" (via 
poll/wait) and the "returncode" (via the .returncode attribute). This is 
good, because it's makes it easy to migrate from the earlier API. 


> 5. I don't like the names fromchild, tochild, and childerr. What's
>    wrong with stdin, stdout, and stderr? OK, stdin is open for write,
>    and the other two for read, but that's fairly obvious as soon as
>    you think about it a bit.

Here's how I see it:

(fromchild, tochild, childerr):
+ Same as with popen2. Easy to migrate etc. 

+ No risk for confusion when connecting the parents stdout to the childs 
  stdin, for example. 

(stdin, stdout, stderr):
+ Nice symmetri with the arguments to the Popen class. 

+ Not as ugly as (fromchild, tochild, childerr)


I need input on this one. I'll change this to whatever people likes best. 


> The biggest issue, though, is that args as a sequence of program
> arguments is very Unix-specific. This is a big incompatibility between
> Unix and Windows - in Windows, the low-level operation (CreateProcess)
> takes a *command line* which is passed unchanged to the child process. 
> The child then parses that command line for itself (often, but not
> always, in the C runtime). In Unix, exec() takes an *argument list*. 
> If you have a command line, you have to split it yourself, a task
> which is usually delegated to /bin/sh in the absence of anything else. 
> This is hard to handle portably.

Oh, I've never thought of this. 


> I'd suggest that the Windows implementation allow two possibilities
> for the "args" argument. If args is a string, it is passed intact to
> CreateProcess. That is the recommended, and safe, approach. The shell
> is still not invoked, so there's no security issue. If a (non-string)
> sequence is passed, the Windows implementation should (for Unix
> compatibility) build a command line, using its best attempt to quote
> things appropriately (this can never be safe, as different programs
> can implement different command line parsing). This is not safe (wrong
> answers are the main problem, not security holes), but will improve
> portability.

I've found some documentation on this.  
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccelng/htm/progs_12.asp
is interesting. It describes how the MS C runtime translates the
commandline to an argv array. Let's call this "Algorithm A". 

When passed a sequence, popen5 should translate this sequence into a
string by using algorithm A backwards. This should definitely be 
implemented in popen5. 

There are two more cases:

1) Should popen5 support a string argument on Windows?

and

2) Should popen5 support a string argument on UNIX?


You seems to have made up your mind about case 1, and even thinks that 
this should be "recommended". I'm not that sure. 

What about case 2 ? This could be supported by converting the string to an 
sequence using Algorithm A. One large problem though is that Algorithm A 
is *not* the same as a typical shell uses. For example, an OS X user might 
want to do:

    Popen("ls '/Applications/Internet Explorer'")

This won't work if we use Algorithm A. 

If we extend Algorithm A to support single quotes as well, this will not 
work as expected on Windows:

    Popen("echo 'hello'")

Sigh. 


> I agree with Guido's comments on python-dev - this module (popen5 is a
> *horrible* name - I'd prefer "process", but am happy if someone comes
> up with a better suggestion) should aim to be the clear "best of
> breed" process control module for all platforms.

The only drawback with "process" is that it is already in use by 
http://starship.python.net/~tmick/#process. 


> I hope this is of some use,

Indeed. 

-- 
/Peter Åstrand <astrand at lysator.liu.se>