[Python-Dev] subprocess not escaping "^" on Windows

[eryk sun]

On Sun, Jan 7, 2018 at 6:48 PM, Christian Tismer <tismer at stackless.com> wrote:
> That is true.
> list2cmdline escapes partially, but on NT and Windows10, the "^" must
> also be escaped, but is not. The "|" pipe symbol must also be escaped
> by "^", as many others as well.
>
> The effect was that passing a rexexp as parameter to a windows program
> gave me strange effects, and I recognized that "^" was missing.
>
> So I was asking for a coherent solution:
> Escape things completely or omit "shell=True".
>
> Yes, there is a list of chars to escape, and it is Windows version
> dependent. I can provide it if it makes sense.

subprocess.list2cmdline is meant to help support cross-platform code,
since Windows uses a command-line instead of an argv array. The
command-line parsing rules used by VC++ (and CommandLineToArgvW) are
the most common in practice. list2cmdline is intended for this set of
applications. Otherwise pass args as a string instead of a list.

In CMD we can quote part of a command line in double quotes to escape
special characters. The quotes are preserved in the application
command line. This can get complicated when we need to preserve
literal quotes in the command line of an application that uses VC++
backslash escaping. CMD doesn't recognize backslash as an escape
character, which gives rise to a quoting conflict between CMD and the
application. Some applications support translating single quotes to
double quotes in this case (e.g. schtasks.exe). Single quotes
generally aren't used in CMD, except in a `for /f` loop, but this can
be forced to use backquotes instead via `usebackq`.

Quoting doesn't escape the percent character that's used for
environment variables. In batch scripts percent can be escaped by
doubling it, but not in /c commands. Some applications can translate a
substitute character in this case, such as "~" (e.g. setx.exe).
Otherwise, we can usually disrupt matching an existing variable by
adding a "^" character after the first percent character. The "^"
escape character gets consumed later on in parsing -- as long as it's
not quoted (see the previous paragraph for complications).
Nonetheless, "^" is a valid name character, so there's still a
possibility of matching an environment variable (perhaps a malicious
one).  For example:

    C:\>python -c "print('"%^"time%')"
    %time%

    C:\>set "^"time=spam"
    C:\>python -c "print('"%^"time%')"
    spam

Anyway, we're supposed to pass args as a string when using the shell
in POSIX, so we may as well stay consistent with this in Windows.
Practically no one wants the resulting behavior when passing a shell
command as a list in POSIX. For example:

    >>> subprocess.call(['echo \\$0=$0 \\$1=$1', 'spam', 'eggs'], shell=True)
    $0=spam $1=eggs

It's common to discourage using `shell=True` because it's considered
insecure. One of the reasons to use CMD in Windows is that it tries
ShellExecuteEx if CreateProcess fails. ShellExecuteEx supports "App
Paths" commands, file actions (open, edit, print), UAC elevation (via
"runas" or if requested by the manifest), protocols (including
"shell:"), and opening folders in Explorer. It isn't a scripting
language, however, so it doesn't pose the same risk as using CMD.
Calling ShellExecuteEx could be integrated in subprocess as a new
Popen parameter, such as `winshell` or `shellex`.