From janssen at parc.com  Mon May 24 16:46:32 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 24 May 2010 07:46:32 PDT
Subject: [stdlib-sig] quoting arguments for command-line evaluation
Message-ID: <78641.1274712392@parc.com>

I'm kind of surprised to see that the subprocess module doesn't have a
documented function for calling to properly quote command-line
arguments.  Nor does it seem to do so automatically -- at least on Unix.

It *does* have an undocumented function called "list2cmdline" (refered
to in passing in the docs as a "method"), which is called automatically
if the "args" parameter to the Popen() initializer is a sequence, and
the platform is Windows, but not otherwise.  No similar quoting of args
is done for Unix.

On the other hand, the "pipes" module has another undocumented function
called "quote", which seems to perform similar quoting, but for sh.
"Pipes" is documented to work only under Unix -- which is a bit strange,
as cmd.exe supports pipes on Windows as well.

Given that Python, even 3, ships with os.system() and os.popen() and
subprocess and pipes and perhaps others that I don't know about,
exposing a quoting mechanism that appropriately quotes for either sh or
cmd.exe, depending on platform, wouldn't be a bad idea.  And perhaps
pipes should be eliminated, or re-written in terms of subprocess to be
cross-platform.

Bill

From ctb at msu.edu  Mon May 24 16:51:30 2010
From: ctb at msu.edu (C. Titus Brown)
Date: Mon, 24 May 2010 07:51:30 -0700
Subject: [stdlib-sig] quoting arguments for command-line evaluation
In-Reply-To: <78641.1274712392@parc.com>
References: <78641.1274712392@parc.com>
Message-ID: <20100524145130.GB3793@idyll.org>

On Mon, May 24, 2010 at 07:46:32AM -0700, Bill Janssen wrote:
> I'm kind of surprised to see that the subprocess module doesn't have a
> documented function for calling to properly quote command-line
> arguments.  Nor does it seem to do so automatically -- at least on Unix.

Bill, I may be about to reveal my profound ignorance here, but...

do you need to quote arguments passed as a list on UNIX?  My impression
was that if shell=False, then it wasn't needed.

> It *does* have an undocumented function called "list2cmdline" (refered
> to in passing in the docs as a "method"), which is called automatically
> if the "args" parameter to the Popen() initializer is a sequence, and
> the platform is Windows, but not otherwise.  No similar quoting of args
> is done for Unix.
> 
> On the other hand, the "pipes" module has another undocumented function
> called "quote", which seems to perform similar quoting, but for sh.
> "Pipes" is documented to work only under Unix -- which is a bit strange,
> as cmd.exe supports pipes on Windows as well.
> 
> Given that Python, even 3, ships with os.system() and os.popen() and
> subprocess and pipes and perhaps others that I don't know about,
> exposing a quoting mechanism that appropriately quotes for either sh or
> cmd.exe, depending on platform, wouldn't be a bad idea.  And perhaps
> pipes should be eliminated, or re-written in terms of subprocess to be
> cross-platform.

Yeah, this whole part of the stdlib is a small horror show.

(I didn't even know the 'pipes' module existed... wow.)

--titus
-- 
C. Titus Brown, ctb at msu.edu

From janssen at parc.com  Mon May 24 18:23:25 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 24 May 2010 09:23:25 PDT
Subject: [stdlib-sig] quoting arguments for command-line evaluation
In-Reply-To: <20100524145130.GB3793@idyll.org>
References: <78641.1274712392@parc.com> <20100524145130.GB3793@idyll.org>
Message-ID: <80028.1274718205@parc.com>

C. Titus Brown <ctb at msu.edu> wrote:

> On Mon, May 24, 2010 at 07:46:32AM -0700, Bill Janssen wrote:
> > I'm kind of surprised to see that the subprocess module doesn't have a
> > documented function for calling to properly quote command-line
> > arguments.  Nor does it seem to do so automatically -- at least on Unix.
> 
> Bill, I may be about to reveal my profound ignorance here, but...
> 
> do you need to quote arguments passed as a list on UNIX?  My impression
> was that if shell=False, then it wasn't needed.

Yes, that's right.  You need to quote them only if you are passing a
whole command line to be evaluated as a shell, as with os.system() or
shell=True and args a string with subprocess.  I think subprocess gets
all this more or less right.

In my case, I was building up a command line to pass as a single string
to subprocess, and needed a way to quote the args explicitly.  That's
why I went looking at this.

Bill

From ianb at colorstudy.com  Mon May 24 19:13:32 2010
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 24 May 2010 12:13:32 -0500
Subject: [stdlib-sig] quoting arguments for command-line evaluation
In-Reply-To: <80028.1274718205@parc.com>
References: <78641.1274712392@parc.com> <20100524145130.GB3793@idyll.org> 
	<80028.1274718205@parc.com>
Message-ID: <AANLkTin7Tx1i8L_LyDd4k5c3lEqdyDR0QGN0nll8Imy1@mail.gmail.com>

On Mon, May 24, 2010 at 11:23 AM, Bill Janssen <janssen at parc.com> wrote:

> C. Titus Brown <ctb at msu.edu> wrote:
>
> > On Mon, May 24, 2010 at 07:46:32AM -0700, Bill Janssen wrote:
> > > I'm kind of surprised to see that the subprocess module doesn't have a
> > > documented function for calling to properly quote command-line
> > > arguments.  Nor does it seem to do so automatically -- at least on
> Unix.
> >
> > Bill, I may be about to reveal my profound ignorance here, but...
> >
> > do you need to quote arguments passed as a list on UNIX?  My impression
> > was that if shell=False, then it wasn't needed.
>
> Yes, that's right.  You need to quote them only if you are passing a
> whole command line to be evaluated as a shell, as with os.system() or
> shell=True and args a string with subprocess.  I think subprocess gets
> all this more or less right.
>
> In my case, I was building up a command line to pass as a single string
> to subprocess, and needed a way to quote the args explicitly.  That's
> why I went looking at this.
>

I've had to do this sort of thing for similar reasons, like calling remote
ssh commands, where a shell command is embedded in a positional argument.  I
implemented my own thing, but the way pipes.quote works would have been
nicer (the implementation could use a couple regexes instead of iterating
over strings though).

-- 
Ian Bicking  |  http://blog.ianbicking.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20100524/63c506b7/attachment.html>

From merwok at netwok.org  Tue May 25 12:16:58 2010
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Tue, 25 May 2010 12:16:58 +0200
Subject: [stdlib-sig] quoting arguments for command-line evaluation
In-Reply-To: <80028.1274718205@parc.com>
References: <78641.1274712392@parc.com> <20100524145130.GB3793@idyll.org>
	<80028.1274718205@parc.com>
Message-ID: <4BFBA39A.4060303@netwok.org>

> In my case, I was building up a command line to pass as a single string
> to subprocess, and needed a way to quote the args explicitly.  That's
> why I went looking at this.
FYI, if you need the reverse operation (string ? list), look at shlex.

Cheers

From eric at trueblade.com  Tue May 25 14:22:50 2010
From: eric at trueblade.com (Eric Smith)
Date: Tue, 25 May 2010 08:22:50 -0400
Subject: [stdlib-sig] quoting arguments for command-line evaluation
In-Reply-To: <4BFBA39A.4060303@netwok.org>
References: <78641.1274712392@parc.com>
	<20100524145130.GB3793@idyll.org>	<80028.1274718205@parc.com>
	<4BFBA39A.4060303@netwok.org>
Message-ID: <4BFBC11A.9000704@trueblade.com>

?ric Araujo wrote:
>> In my case, I was building up a command line to pass as a single string
>> to subprocess, and needed a way to quote the args explicitly.  That's
>> why I went looking at this.
> FYI, if you need the reverse operation (string ? list), look at shlex.

Keep in mind that these routines typically only work on Unix-like 
systems, where the shell has a set of well defined rules for quoting and 
parsing. On Windows, each executable is in charge of parsing its own 
parameters from a string. This is usually (but certainly not always) 
done by the C runtime startup. But different runtimes have different 
rules. Sadly, you sometimes have to know which executable you're calling 
before you can know how to escape its arguments.

-- 
Eric.

From p.f.moore at gmail.com  Tue May 25 23:54:18 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 25 May 2010 22:54:18 +0100
Subject: [stdlib-sig] quoting arguments for command-line evaluation
In-Reply-To: <4BFBC11A.9000704@trueblade.com>
References: <78641.1274712392@parc.com> <20100524145130.GB3793@idyll.org>
	<80028.1274718205@parc.com> <4BFBA39A.4060303@netwok.org>
	<4BFBC11A.9000704@trueblade.com>
Message-ID: <AANLkTinq16528roI0xkjsu_fsESqkPrqlP7Ex7OKty1d@mail.gmail.com>

On 25 May 2010 13:22, Eric Smith <eric at trueblade.com> wrote:
> Keep in mind that these routines typically only work on Unix-like systems,
> where the shell has a set of well defined rules for quoting and parsing. On
> Windows, each executable is in charge of parsing its own parameters from a
> string. This is usually (but certainly not always) done by the C runtime
> startup. But different runtimes have different rules. Sadly, you sometimes
> have to know which executable you're calling before you can know how to
> escape its arguments.

However, on Windows the rules used by the C runtime are a pretty good
baseline to work to. That's what subprocess does, and it's a good
practical solution. It's not perfect, but practicality beats purity
and all that....

After all, there are many different shells on Unix, and I suspect an
expert could come up with a suitably nasty set of corner cases where
they are inconsistent. But frankly, it doesn't matter. It matters
somewhat more on Windows, but that's more because the "standard" rules
aren't very user-friendly than because you'll get a lot of breakage.

I'd say that this is a classic case where a library encapsulating
cross-platform differences [1] is extremely useful. If the stdlib
contained a cross-platform shell-quoting module, I'm pretty sure it
could replace a big chunk of adhoc application code.

Paul.

[1] At the OS call level, Unix works with an array of arguments, where
Windows works with a command line. No cross-platform library can
completely hide such a basic distinction. But again, practicality
beats purity...

From pjenvey at underboss.org  Wed May 26 00:37:31 2010
From: pjenvey at underboss.org (Philip Jenvey)
Date: Tue, 25 May 2010 15:37:31 -0700
Subject: [stdlib-sig] quoting arguments for command-line evaluation
In-Reply-To: <AANLkTinq16528roI0xkjsu_fsESqkPrqlP7Ex7OKty1d@mail.gmail.com>
References: <78641.1274712392@parc.com> <20100524145130.GB3793@idyll.org>
	<80028.1274718205@parc.com> <4BFBA39A.4060303@netwok.org>
	<4BFBC11A.9000704@trueblade.com>
	<AANLkTinq16528roI0xkjsu_fsESqkPrqlP7Ex7OKty1d@mail.gmail.com>
Message-ID: <163D4B34-8DE5-44C2-AAB7-2CC13ACDDD10@underboss.org>


On May 25, 2010, at 2:54 PM, Paul Moore wrote:

> On 25 May 2010 13:22, Eric Smith <eric at trueblade.com> wrote:
>> Keep in mind that these routines typically only work on Unix-like systems,
>> where the shell has a set of well defined rules for quoting and parsing. On
>> Windows, each executable is in charge of parsing its own parameters from a
>> string. This is usually (but certainly not always) done by the C runtime
>> startup. But different runtimes have different rules. Sadly, you sometimes
>> have to know which executable you're calling before you can know how to
>> escape its arguments.
> 
> However, on Windows the rules used by the C runtime are a pretty good
> baseline to work to. That's what subprocess does, and it's a good
> practical solution. It's not perfect, but practicality beats purity
> and all that....

> I'd say that this is a classic case where a library encapsulating
> cross-platform differences [1] is extremely useful. If the stdlib
> contained a cross-platform shell-quoting module, I'm pretty sure it
> could replace a big chunk of adhoc application code.

I'd also like to see both uses, splitting and joining posix/windows command lines cleaned up into a separate module.

The interface to shlex can be a little daunting to deal with, with it's odd non-posix mode that isn't actually for the Windows shell. Plus it's complicated by the fact that it's also intended to be used as a generic parser.

There's already a ticket requesting the ability to parse a Windows command line: http://bugs.python.org/issue1724822

--
Philip Jenvey