PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API

[Chris Angelico]

On Thu, May 7, 2015 at 1:33 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> On Thursday 07 May 2015 12:19, Chris Angelico wrote:
>
>> On Thu, May 7, 2015 at 10:58 AM, Dave Angel <davea at davea.name> wrote:
>>> There's nothing Windows-specific about that behaviour.  In Linux, there
>>> are
>>> bash commands that can only be run by using shell=True.  Fortunately
>>> Popen didn't make the mistake of pretending it's a shell.
>>
>> But bash commands aren't the same as shell scripts. For instance, if
>> you want to enumerate bash aliases, you can't exec() to the 'alias'
>> command, because there isn't one. But shell scripts *can* be exec'd:
>
> Um, are we still talking about Python here? exec("alias") fails with
> NameError on all the versions of Python I know. *semi-wink*
>
> I'm guessing you're taking about some other exec, it might be a good idea
> that on a Python mailing list you don't assume that we're all going to
> understand the context :-)

Fair point. I'm talking about the underlying exec* family of
functions, which are used on Unix-like systems to do normal process
execution. More on that later.

>> $ grep $ exec_demo.*
>> exec_demo.c:#include <stdio.h>
>> exec_demo.c:#include <unistd.h>
>> exec_demo.c:int main()
>> exec_demo.c:{
>> exec_demo.c: printf("This part is coming from C code.\n");
>> exec_demo.c: int err=execl("./exec_demo.sh", 0);
>> exec_demo.c: printf("exec() failed! %d\n",err);
>> exec_demo.c:}
>> exec_demo.sh:#!/bin/sh
>> exec_demo.sh:echo This part ran from the shell.
>> exec_demo.sh:echo Hello, world!
>> $ ./a.out
>> This part is coming from C code.
>> This part ran from the shell.
>> Hello, world!
>
>> $ pike -e 'Process.exec("./exec_demo.sh");'
>> This part ran from the shell.
>> Hello, world!
>
> Okay, so C code can call the shell. So can Pike.
>
>
>> $ python -c 'import subprocess; subprocess.call(["./exec_demo.sh"])'
>> This part ran from the shell.
>> Hello, world!
>
> And so can Python. I'm not entirely sure what point you are trying to make
> here, or how it relates to the OP's problem that when he calls
>
> subprocess.Popen(['foo'])
>
> he expects it to run any of foo.exe, foo.cmd, foo.bat (and possibly any
> other number of executable files). Are you agreeing with him or disagreeing?

I'm stating that this works on Unix with shell=False. Consequently,
the OP's request that it work thusly on Windows ought to be possible,
and ought to be reasonable.

> Apart from any other number of problems, surely having "foo" alone run
> foo.exe, foo.bat etc. is at best confusing and at worst a security risk?
> What if you have *both* foo.exe and foo.bat in the same directory?

There's a specific search order. Back in the days of DOS, it was
simply "com, then exe, then bat", but on modern Windowses, I think
it's governed by an environment variable (similarly to PATH for the
directories to search in). It's identical security risk to the
possibility of putting something earlier in $PATH; if you insist on
running a specific executable, you put the entire path, and then
there's no problem. (Actually, I'd consider this to be a feature, not
a bug - it's the equivalent of shadowing built-ins in a Python module.
When you want it, it's really useful.)

>> (Python doesn't seem to have any way to 'exec', but a subprocess comes
>> to the same thing.)
>
> According to `man exec` on my Linux system, I don't think that is correct.
> The exec* family of functions "replaces the current process image with a new
> process image", they don't run in a subprocess.
>
> I think the Python equivalent of Unix exec* commands are the various
> os.exec* functions.

Oh, I forgot about os.exec. That's a better equivalent:

$ python3 -c 'import os; os.execl("./exec_demo.sh","exec_demo.sh")'
This part ran from the shell.
Hello, world!

When you call subprocess.* to run something, what happens is generally
that Python forks and exec's to the new process. The parent of the
fork keeps running (or maybe waits on the child immediately), the
child uses one of the exec family of functions to replace itself with
the new process. This is different from Windows, where there's a
standard API function for "start this program in a new process".
Advantages of the fork/exec model include that you can do stuff in
between the two steps - for instance, you fork, then you change your
user credentials, root directory, current directory, nice value, usage
limits, etc, etc, etc, etc, prior to exec'ing. Unix systems don't need
a way to say "run this process with this root directory", because it
can all be built up from primitives.

So yes, in terms of the ability to locate other executables, it makes
no difference whether you fork first or not - the exec call is the
same.

> Hmmm. I'm not sure if this is relevant, or if I'm going off on a tangent,
> but if I write a short bash script and set the execute permission:
>
> steve at runes:~$ chmod u+x test.sh
> steve at runes:~$ cat test.sh
> echo "Running shell script"
>
> subprocess.call fails unless I set shell=True:
>
> py> p = subprocess.Popen('./test.sh', shell=True)
> py> Running shell script
>
> py> p = subprocess.Popen('./test.sh', shell=False)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/local/lib/python2.7/subprocess.py", line 711, in __init__
>     errread, errwrite)
>   File "/usr/local/lib/python2.7/subprocess.py", line 1308, in
> _execute_child
>     raise child_exception
> OSError: [Errno 8] Exec format error
>
>
> How is this any different from needing to specify shell=True for .bat and
> .cmd files under Windows? This is not a rhetorical question, I actually want
> to know.

Hmm... hm... Ha! Found the difference. I had an explicit shebang on my
script; yours just starts out with shell commands. That means that
your shell script wasn't truly executable, and thus requires a shell
to execute it. Try adding "#!/bin/sh" to the top and rerun that - at
that point, it becomes kernel-executable instead of just
shell-executable.

ChrisA