PEP 324: popen5 - New POSIX process module

[Moore, Paul]

From: Peter Astrand [mailto:astrand at lysator.liu.se]
>> The biggest issue, though, is that args as a sequence of program
>> arguments is very Unix-specific. This is a big incompatibility between
>> Unix and Windows - in Windows, the low-level operation (CreateProcess)
>> takes a *command line* which is passed unchanged to the child process. 
>> The child then parses that command line for itself (often, but not
>> always, in the C runtime). In Unix, exec() takes an *argument list*. 
>> If you have a command line, you have to split it yourself, a task
>> which is usually delegated to /bin/sh in the absence of anything else. 
>> This is hard to handle portably.

> Oh, I've never thought of this. 

It's a major issue. I've been porting process-creation code from Unix to
Windows for a long while now, and there simply isn't a good, compatible,
answer.

> I've found some documentation on this.  
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccelng/htm/progs_12.asp
> is interesting. It describes how the MS C runtime translates the
> commandline to an argv array. Let's call this "Algorithm A". 

The problem here is that not all programs use the MSVC runtime.
Borland C uses a subtly different algorithm, and programs written in
other languages often don't use an argv concept at all.

Even adding quotes when not necessary can cause certain programs to
fail.

> When passed a sequence, popen5 should translate this sequence into a
> string by using algorithm A backwards. This should definitely be 
> implemented in popen5. 

> There are two more cases:
>
> 1) Should popen5 support a string argument on Windows?
>
> and
>
> 2) Should popen5 support a string argument on UNIX?


> You seems to have made up your mind about case 1, and even thinks that 
> this should be "recommended". I'm not that sure. 

Using a list works most of the time, but can break in subtle, and surprising
ways. I have coded "Algorithm A in reverse" a number of times, and *always*
managed to construct a program that "broke". With Windows GUI programs
(not the key use case for this module, I admit) it's not remotely hard to
construct a broken example :-(

> What about case 2 ? This could be supported by converting the string to an 
> sequence using Algorithm A. One large problem though is that Algorithm A 
> is *not* the same as a typical shell uses. For example, an OS X user might 
> want to do:
>
>    Popen("ls '/Applications/Internet Explorer'")
>
> This won't work if we use Algorithm A. 
>
> If we extend Algorithm A to support single quotes as well, this will not 
> work as expected on Windows:
>
>    Popen("echo 'hello'")
>
> Sigh.

Sigh indeed. It's just not cross-platform no matter how you try. BTW, your
echo example won't work in any case on Windows, as "echo" is a shell
builtin, and not available as a standalone executable. So on Windows,
you'd need

    Popen("cmd /c echo hello")

And don't get me started on how cmd/c's quoting peculiarities can make this
even worse :-(

> The only drawback with "process" is that it is already in use by 
> http://starship.python.net/~tmick/#process. 

Ah. I wasn't aware of that module. I wonder whether Trent would be willing
to combine his code and yours somehow, and donate the name?

Paul