From marklists@mceahern.com  Tue Feb 12 20:01:41 2002
From: marklists@mceahern.com (Mark McEahern)
Date: Tue, 12 Feb 2002 12:01:41 -0800
Subject: [getopt-sig] optik tastes great, less filling
Message-ID: <NCBBLFCOHHDIKCAFGCFBKEHPKCAA.marklists@mceahern.com>

I haven't joined the list, so I won't see responses, but I wanted to forward
this feature request I sent to python-dev:

-----Original Message-----
From: Mark McEahern [mailto:marklists@mceahern.com]
Sent: Monday, February 11, 2002 11:17 AM
To: python-list@python.org
Cc: python-dev@python.org
Subject: RE: Option Parsing Libraries


[Paul Prescod]
> If you have a competitive library, or suggestions for changes to Optik,
> please forward your comments to python-dev mailing list
> (python-dev@python.org).

I love optik.  We use it for all of our option parsing.

I have one feature request related to error handling.  I've attached sample
code below that shows a common thing I end up doing:  raising an error if a
required option is missing.  I guess it's not really an option, then, is it?
<wink>  Anyway, I searched optik's documentation for some way to "inspect"
the options collection itself for the original information used when
creating the option.  In the code below, you'll notice the requiredVar
method takes a description parameter.  It would be nice to be able to do
something like this instead:

	if var is None:
		parser.error("Missing: %s" % options.var.description)

In fact, it would seem that this itself is so common it would be "built-in"
to optik itself.  So that all I have to do is declare an option as required
when I add it with add_option?

Thanks,

// mark

#! /usr/bin/env python
# testError.py

from optik import OptionParser

def requiredVar(parser, options, var, description):
    """Raise a parser error if var is None."""
    if var is None:
        # Here's where it'd be nice to have access to the attributes of the
        # options; at the very least, so I could say which option is missing
        # without having to pass in the description.
        parser.error("Missing: %s" % description)

def parseCommandLine():
    """Parse the command line options and return (options, args)."""

    usage = """usage: %prog [options]
Testing optik's error handling.
"""
    parser = OptionParser(usage)
    parser.add_option("-f", "--file", type="string", dest="filename",
                      metavar="FILE", help="read data from FILE",
default=None)

    options, args = parser.parse_args()

    requiredVar(parser, options, options.filename, "filename")

def main():
    options, args = parseCommandLine()

if __name__=="__main__":
    main()



From corbin@samsix.com  Tue Feb 12 21:16:48 2002
From: corbin@samsix.com (Thomas R. Corbin)
Date: Tue, 12 Feb 2002 16:16:48 -0500
Subject: [getopt-sig] Optik is a wonderful piece of code.
Message-ID: <200202122116.g1CLGmr16065@laptop>

	I've fallen in love with using optik and in fact, optik is what has 
convinced my team to switch some code to python.    It's been a real team 
builder, playing with it.    I know it seems silly to like a piece of code so 
much, but we all do.    It would be wonderful it it were the basis of a new 
standard python module.    Optik has made command line processing so much 
simpler than just about any other code except libcmdline in C++.


From tim.one@comcast.net  Tue Feb 12 21:13:38 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 12 Feb 2002 16:13:38 -0500
Subject: [getopt-sig] RE: [Python-Dev] a different approach to argument parsing
In-Reply-To: <a3ed7e8ff9854129ba52c84e0d774286@plan9.bell-labs.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCELGNLAA.tim.one@comcast.net>

[Russ Cox, on the Plan9-ish approach]
> ...
> For concreteness, I'd much rather write:
>
> 	if o=='-n' or o=='--num':
> 		ncopies = opt.optarg(opt.needtype(int))

If a commandline has

    -abc

then can you-- with only finite pain --manage to treat it as if it were

    -a -b -c

?  Or doesn't this scheme support collapsing multiple single-letter
arguments (where "support" == "finite pain")?  If the option parser isn't
told in advance what the possible options are, it seems like it couldn't
help in this case.  Or is it that Plan 9 doesn't support options with names
longer than one character?

> ...
> [This is documented at
> http://plan9.bell-labs.com/magic/man2html/2/ARGBEGIN,
> for anyone who is curious.]

I tried, but that link was a 404 for me.



From joshua_rodman@yahoo.com  Tue Feb 12 22:17:52 2002
From: joshua_rodman@yahoo.com (Joshua Rodman)
Date: Tue, 12 Feb 2002 14:17:52 -0800 (PST)
Subject: [getopt-sig] Prefer non-imperative -- my 1 cent
In-Reply-To: <E16akej-0005WM-00@mail.python.org>
Message-ID: <20020212221752.25586.qmail@web12703.mail.yahoo.com>

I don't have a nice clear argument here, sorry.

I feel that the classic option walking loop (such as old-style getopt,
or the plan9 macro system) is prone to messyness as compared to a set 
of option elements such as Optik. 

In my (limited) experience of creationg various option parsers for 
different projects, I have found that the logic of the loop tends to 
get sufficiently bloated that it is difficult to take in the structure 
of it at once.  

The Optik way of thinking about things tries to tie all the logic related 
to an option into a single solitary stanza, complete with callbacks if 
(rarely) necessary. It may not be easier to see all the option parsing code,
but I believe it is definitely easier to see what is going on with a 
particular option.  I think this is important. When enhancing a program, 
I want to think about the one option, not the entire machinery.

I feel this makes the possibly wordier setup code simpler in concept, and
thus easier to use and grasp.  Contrarily, I think the main advantage of
the traditional approach is that it is traditional.  That is, people coming
to code done the way they are used to will likely grasp it much more quickly.


Yes, this is a touchy-feely description, without much hard currency.
But please do consider it, I think I'm onto something important.

-josh

__________________________________________________
Do You Yahoo!?
Send FREE Valentine eCards with Yahoo! Greetings!
http://greetings.yahoo.com


From rsc@plan9.bell-labs.com  Tue Feb 12 22:43:52 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Tue, 12 Feb 2002 17:43:52 -0500
Subject: [getopt-sig] Prefer non-imperative -- my 1 cent
Message-ID: <4bdedb6dc1db5fee2756e63d26ecb6b5@plan9.bell-labs.com>

Can you give an example (perhaps a big one) where option
parsing with Optik-style objects makes things clearer
than the traditional loop?  

I will grant that perhaps there are cases where the
loop might just get too big, but I have a hard time
envisioning them.  In particular, just having
lots of options doesn't seem like enough to make
the option loop not work.  You claim that tying "all
the logic related to an option into a single solitary
stanza" is a good thing, which I grant.  What I don't
understand is how many of these:

	elif o=='-o' or '--optionlong':
		# single solitary stanza here

don't actually do that or could get "sufficiently
bloated that it is difficult to take in the structure
of it at once."

All that aside, my big problem with the Optik-style
modules, and to a lesser extent getopt, is the barrier
to entry.  Option parsing is something everyone has to
do, more or less; it should require as little work
as possible to get simple parsing in place.  The fact
that it's _not_ simple enough is why people settle for
quick and dirty things like

	if sys.argv[1]=='-d':
		debug=1
		sys.argv = sys.argv[1:]

when they want just one or two options.  This problem
is certainly endemic in C programs; I haven't surveyed
enough Python programs to make informed comments about
how widespread the practice is in Python.

On the other hand, since (in the module I posted)
it's hardly any more code to use the option parser:

	opt = OptionParser()
	for o in opt:
		if o=='-d':
			debug=1

I think people are more likely to use it, even for quick
and dirty argument parsing.  I think that's the big deal.

Of course, it's possible that there is some hybrid scheme
that would satisfy everyone, but I really don't have
any feel for argument parsing so hairy that you need
parsing objects and callbacks and the like: someone please
enlighten me.

Thanks.
Russ


From joshua_rodman@yahoo.com  Tue Feb 12 23:15:08 2002
From: joshua_rodman@yahoo.com (Joshua Rodman)
Date: Tue, 12 Feb 2002 15:15:08 -0800 (PST)
Subject: [getopt-sig] Prefer non-imperative -- my 1 cent
In-Reply-To: <840a061321eebdeff2ed4028630413dc@plan9.bell-labs.com>
Message-ID: <20020212231508.34102.qmail@web12703.mail.yahoo.com>

--0-1006439860-1013555708=:30853
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

--- Russ Cox <rsc@plan9.bell-labs.com> wrote:

> I will grant that perhaps there are cases where the
> loop might just get too big, but I have a hard time
> envisioning them.  In particular, just having
> lots of options doesn't seem like enough to make
> the option loop not work.  You claim that tying "all
> the logic related to an option into a single solitary
> stanza" is a good thing, which I grant.  What I don't
> understand is how many of these:
> 
> 	elif o=='-o' or '--optionlong':
> 		# single solitary stanza here
> 
> don't actually do that or could get "sufficiently
> bloated that it is difficult to take in the structure
> of it at once."

In practice I find that the tradditional style equates to:

initialize a bunch of things to defaults
call the parser
loop:
   handle option:
      verify option is valid
         if not: usage()
      ...

perform some additional checks and assignments

If nothing else, optik-style puts the initializers, the variable 
assigned to, and the break on invalid assignment all in one spot,
instead of sprinkled over three locations. 

On a possibly incorrect side-note, I've found that the traditional
loop seems to encourage the use of state flags during the walking
of the loop.  At least, in larger projects I've bumped into, this
seems rather common, and rather ugly.  I'm speaking here from the
school of thought that says that constricture which eliminates
bad choices is good.  Some people hate this school of thought. :-)

> All that aside, my big problem with the Optik-style
> modules, and to a lesser extent getopt, is the barrier
> to entry.  Option parsing is something everyone has to
> do, more or less; it should require as little work
> as possible to get simple parsing in place.  The fact
> that it's _not_ simple enough is why people settle for
> quick and dirty things like
> 
> 	if sys.argv[1]=='-d':
> 		debug=1
> 		sys.argv = sys.argv[1:]
> 
> when they want just one or two options.  This problem
> is certainly endemic in C programs; I haven't surveyed
> enough Python programs to make informed comments about
> how widespread the practice is in Python.
> 
> On the other hand, since (in the module I posted)
> it's hardly any more code to use the option parser:
> 
> 	opt = OptionParser()
> 	for o in opt:
> 		if o=='-d':
> 			debug=1
> 
> I think people are more likely to use it, even for quick
> and dirty argument parsing.  I think that's the big deal.

WARNING: Anecdotal consideration ahead!

You're right of course.  Traditional getopt was too much of
a barrier to use for me. Strangely, the ornateness of Optik
I found much easier to get into.  Scripts I'd never have 
bothered to set up getopt for, I'm finding I bother to create 
option arguments for. I think it has a lot to do with the 
elimination of the usage() function, the usage string, and 
the copious calls to that exit-point from the program.

Maybe I just haven't given a system like yours a proper try.
I admit I'm biased against it: eliminating the loop walking 
is _very_ seductive to me. I don't like the idea of building 
the loop machinery every time when that's the same across 
all option parsers.

> Of course, it's possible that there is some hybrid scheme
> that would satisfy everyone, but I really don't have
> any feel for argument parsing so hairy that you need
> parsing objects and callbacks and the like: someone please
> enlighten me.

Parsing objects don't seem heavy to me.  Callbacks are of course,
but I've not yet felt the desire to use them.  What _has_ been
a huge benefit to me is the extensibility of optik. 

I had a tool which required option pairs for a left hand side
and a right hand side of a set of operations.  For example, 
given a fruit-eating program, you want to be able to say:

  eatfruits --fruitcount 5 

and have it eat 5 apples and five pears.  Or

  eatfruits --pears --fruitcount 5 --apples --fruitcount 3

and have it eat 3 apples and 5 pears.  This example stinks, in
the real case it made sense to do this.  I had about 10 different
values with more expected in the future, some of them dependent on
each other.  Creating this in a traditional loop structure would
have me create a tristate flag variable and extra handling with
each assignment.  I _think_ it would have been painful to build.
I was avoiding it because it sounded just awful, even to the point
of having a poor framework for debugging the rest of the system.

In optik, I subclassed the option object, added my behavior, 
and it was done.  Admittedly I still had a tristate flag variable,
but the logic was all encapsulated in the interface.  

The actual example is a bit of a counterargument, as I'm extending
in a direction it isn't really designed to go, so it's a bit bulky.
I'm also a rather inexpert Python programmer, so I suspect I'm 
implementing passthrough to the base class inexpertly.  Attached
if interesting.

All this may only serve to elaborate on my prejudice, as I haven't
thought deeply enough about the other way.  Feel free to dump it in
the trash if it says nothing useful to you.

> Thanks.
> Russ

Thanks for taking my rambling more seriously than it really deserves. :-)

-josh



__________________________________________________
Do You Yahoo!?
Send FREE Valentine eCards with Yahoo! Greetings!
http://greetings.yahoo.com
--0-1006439860-1013555708=:30853
Content-Type: application/octet-stream; name="optik_pairs.py"
Content-Transfer-Encoding: base64
Content-Description: optik_pairs.py
Content-Disposition: attachment; filename="optik_pairs.py"

IyEvdXNyL2Jpbi9lbnYgcHl0aG9uCmZyb20gb3B0aWsub3B0aW9uX3BhcnNl
ciBpbXBvcnQgVmFsdWVzCmZyb20gb3B0aWsgaW1wb3J0IE9wdGlvbgoKY2xh
c3MgUGFpck9wdGlvbihPcHRpb24pOgogICAgQUNUSU9OUyA9IE9wdGlvbi5B
Q1RJT05TICsgICAgICAgICAgICAgKCJzdG9yZV9wYWlyIiwpCiAgICBTVE9S
RV9BQ1RJT05TID0gT3B0aW9uLlNUT1JFX0FDVElPTlMgKyAoInN0b3JlX3Bh
aXIiLCkKICAgIFRZUEVEX0FDVElPTlMgPSBPcHRpb24uVFlQRURfQUNUSU9O
UyArICgic3RvcmVfcGFpciIsKQoKICAgIE1FVEFfU1RPUkVfQUNUSU9OUyA9
ICAgICAgICAgICAgICAgICAgICgic3RvcmVfcGFpciIsKQoKICAgIEFUVFJT
ID0gT3B0aW9uLkFUVFJTICsgICAgICAgICAgICAgICAgIFsic3ViYWN0aW9u
Il0KCiAgICBkZWYgX19pbml0X18oc2VsZiwgKm9wdHMsICoqYXR0cnMpOgog
ICAgICAgIGFwcGx5KE9wdGlvbi5fX2luaXRfXywgKHNlbGYsKSArIG9wdHMs
IGF0dHJzKQogICAgICAgIHNlbGYuX2NoZWNrX3N1YmFjdGlvbigpCgogICAg
ZGVmIF9jaGVja19zdWJhY3Rpb24oc2VsZik6CiAgICAgICAgaWYgc2VsZi5h
Y3Rpb24gaW4gc2VsZi5NRVRBX1NUT1JFX0FDVElPTlM6CiAgICAgICAgICAg
IGlmIHNlbGYuc3ViYWN0aW9uIGlzIE5vbmU6CiAgICAgICAgICAgICAgICBz
ZWxmLnN1YmFjdGlvbiA9ICJzdG9yZSIKICAgICAgICAgICAgZWxpZiBzZWxm
LmFjdGlvbiBub3QgaW4gc2VsZi5TVE9SRV9BQ1RJT05TOgogICAgICAgICAg
ICAgICAgcmFpc2UgT3B0aW9uRXJyb3IoImludmFsaWQgc3ViYWN0aW9uOiAl
ciIgJSBzZWxmLnN1YmFjdGlvbiwgCiAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICBzZWxmKQoKICAgIGRlZiB0YWtlX2FjdGlvbihzZWxmLCBh
Y3Rpb24sIGRlc3QsIG9wdCwgdmFsdWUsIHZhbHVlcywgcGFyc2VyKToKICAg
ICAgICBpZiBhY3Rpb24gPT0gInN0b3JlX3BhaXIiOgogICAgICAgICAgICB0
YXJnX2xvYyA9IHZhbHVlcy5lbnN1cmVfdmFsdWUoZGVzdCwgW05vbmVdKjIp
CiAgICAgICAgICAgIGR1bW15dmFsdWVzID0gVmFsdWVzKCkKICAgICAgICAg
ICAgaWYgbm90IHZhbHVlcy5lbnN1cmVfdmFsdWUoJ3BhaXJfY29udHJvbCcs
ICcnKSA9PSAic2Vjb25kIjoKICAgICAgICAgICAgICAgIGR1bW15dmFsdWVz
LmZpcnN0ID0gdGFyZ19sb2NbMF0KICAgICAgICAgICAgICAgIHNlbGYudGFr
ZV9hY3Rpb24oc2VsZi5zdWJhY3Rpb24sICdmaXJzdCcsIG9wdCwgdmFsdWUs
IAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkdW1teXZhbHVl
cywgcGFyc2VyKQogICAgICAgICAgICAgICAgdGFyZ19sb2NbMF0gPSBkdW1t
eXZhbHVlcy5maXJzdAogICAgICAgICAgICBpZiBub3QgdmFsdWVzLnBhaXJf
Y29udHJvbCA9PSAiZmlyc3QiOgogICAgICAgICAgICAgICAgZHVtbXl2YWx1
ZXMuc2Vjb25kID0gdGFyZ19sb2NbMV0KICAgICAgICAgICAgICAgIHNlbGYu
dGFrZV9hY3Rpb24oc2VsZi5zdWJhY3Rpb24sICdzZWNvbmQnLCBvcHQsIHZh
bHVlLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZHVtbXl2
YWx1ZXMsIHBhcnNlcikKICAgICAgICAgICAgICAgIHRhcmdfbG9jWzFdID0g
ZHVtbXl2YWx1ZXMuc2Vjb25kCiAgICAgICAgZWxzZToKICAgICAgICAgICAg
T3B0aW9uLnRha2VfYWN0aW9uKHNlbGYsIGFjdGlvbiwgZGVzdCwgb3B0LCB2
YWx1ZSwgdmFsdWVzLCBwYXJzZXIpCgo=

--0-1006439860-1013555708=:30853--


From derek@chocolate-fish.com  Wed Feb 13 01:23:04 2002
From: derek@chocolate-fish.com (Derek Harland)
Date: Wed, 13 Feb 2002 01:23:04 -0000
Subject: [getopt-sig] Another Object Orientated Option Parser
Message-ID: <007601c1b42c$fd5129c0$080a0a0a@g7n3g0>

This is a multi-part message in MIME format.

------=_NextPart_000_0073_01C1B42C.FC76F660
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

Greg and other option parsing fiends !!

Here is an option parsing library you may have missed while comparing whats
available.  Its mine, and the reason you missed it is probably because you
weren't sifting though my particular hard drive ;-)

It might bear comparison though.  I'm not suggesting it as an alternative,
but it seems to have most of the features of Optik and maybe useful to you.
In fact it seems equivalent in its features but probably older [does that
make it prior art :-) ].  Its existed for quite some time before the more
recent rash of different ones became available on parnassus and has more
than suited our needs.  As a result, I can say I have never tried any of the
others and am thus completely unqualified to compare them ;-)

Its been used for over two years in a production environment now, which
means
* its stable --- my CVS says I havent touched it in over 12 months (apart
from a change to the DateTime import to shift it to mx)
* it does everything we need, of course, maybe thats because we tailor our
code to its needs ;-)
* it works in a production environment that doesnt tolerate failure (do any
?)
* its 1.5.2 compatible :-)
* its damn easy to use

It is open source. I think its useful to others.  I have never released it
to the world because I have never had the energy to put it up on the web !
And also, I'm never happy with what I write so struggle to release it to the
world ;-). This module is also written in a different style to how I write
today, which just makes it even more painful !

But it is
* Object orientated.
* Allows default values (None by default), or required values
* Enforces typing, ie options are declared to be IntegerOptions or
ComplexListOptions or DateOptions or even EvalOptions.
* completely powered [and thus limited by getopt], as I couldn't be bothered
writing the tokenisation.
* Helpful, literally ;-).  If you don't create a help option then the parser
assumes you are brain dead and creates it for you !  I think this is a
compulsory feature, others may disagree ... I rate it next to white space.
Also, if you dont give an option marked as required it will dump help to
you.

In its base implementation:
* it does not have callbacks (but you can inherit and override processOption
... seen it done although I have never needed it)
* you cannot modify the option structure during parse time (eg making some
options dependent on the existence or value of others).
* only accepts posix/getopt style options.  ie no custom prefixes like +

Sorry this mail is getting long, but then its serving as the documentation
too ;-).

*** Examples of use ***
1. Invoking a script that uses option.py with the --help argument to get
some help ! [the script happens to be option.py in this case, what a
coincidence]
des@fish:tmp>python option.py --help
You're testing the option parser !!
The following options are recognised:
-i <value>, --integer=<value>
   An Integer
   Default is: 4
--list=<value>
   A string list
-l <value>, --list2=<value>
   A required string list
   This Option is Required.
-e <value>, --eval=<value>
   Evaluated expression
-b, --boolean
   An on/off option
-f <value>, --floats=<value>
   A listof floats
   Default is: [2.0, 4.0999999999999996]
-s <value>, --string=<value>
   A string
   This Option is Required.
-h, --help
   Provides this message.

(Note: The argument -- terminates option processing, remaining arguments
will be unparsed.)

2.  Okay, here is the result of running a script that uses option.py.  This
test script (actually option.py itself as it happens) will just print back
the option values you gave.

des@fish:tmp>python option.py -l spam,spam --string="I say!" -e
"[1,2,3]*2" -f 6,9,1.2
Here are the option values after the parse
-i <value>, --integer=<value> has value 4 of type <type 'int'>
--list=<value> has value None of type <type 'None'>
-l <value>, --list2=<value> has value ['spam', 'spam'] of type <type 'list'>
-e <value>, --eval=<value> has value [1, 2, 3, 1, 2, 3] of type <type
'list'>
-b, --boolean has value 0 of type <type 'int'>
-f <value>, --floats=<value> has value [6.0, 9.0, 1.2] of type <type 'list'>
-s <value>, --string=<value> has value I say! of type <type 'string'>
-h, --help has value 0 of type <type 'int'>

Is this helpful to you ?  I have been meaning to rewrite it for a looong
time to make it even more powerful but frankly, its always served needs and
so it hasn't been worth it.

I've attached the code.  Its one module of ~400 lines including doc strings
so its reasonably compact.

Des.

=============================
As a comparison to Optik

> Well, here's what I like about Optik:
>
>   * it ties short options and long options together, so once you
>     define your options you never have to worry about the fact that
>     -f and --file are the same

ditto

>   * it's strongly typed: if you say option --foo expects an int,
>     then Optik makes sure the user supplied a string that can be
>     int()'ified, and supplies that int to you

ditto

>   * it automatically generates full help based on snippets of
>     help text you supply with each option

ditto

>   * it has a wide range of "actions" -- ie. what to do with the
>     value supplied with each option.  Eg. you can store that value
>     in a variable, append it to a list, pass it to an arbitrary
>     callback function, etc.

Not quite.  In option.py, a list is another "type" of option
which is constructed out of eg --words=greg,ward

Callbacks are not provided in the basic option class.
Implementing them though is just a matter of inheriting from Option and
overriding processOption.  I've been aware its possible but have never
bothered with it myself.

>   * you can add new types and actions by subclassing -- how to
>     do this is documented and tested

ditto, except for the documentation ;-)

>   * it's dead easy to implement simple, straightforward, GNU/POSIX-
>     style command-line options, but using callbacks you can be as
>     insanely flexible as you like

option.py only supports getopt style options because thats what it leverages
off !
[or is held back by ...].

>   * provides lots of mechanism and only a tiny bit of policy (namely,
>     the --help and (optionally) --version options -- and you can
>     trash that convention if you're determined to be anti-social)

enforces help options.  If you don't make it will do it for you !



------=_NextPart_000_0073_01C1B42C.FC76F660
Content-Type: text/plain;
	name="option.py"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="option.py"

# system imports=0A=
import sys, string, types, operator=0A=
import getopt, formatter, cStringIO=0A=
=0A=
class OptionError(ValueError):=0A=
    "An error in an option, due to conversion or lack of argument etc."=0A=
    pass=0A=
=0A=
class ParserError(OptionError):=0A=
    "An error occurring during the parsing of command line arguments."=0A=
    pass=0A=
=0A=
class HelpError(OptionError):=0A=
    "An exception indicating Help was requested and provided."=0A=
    pass=0A=
=0A=
class Option:=0A=
    """The base class for options

    This class also models Boolean options, ie options that either exist =
(have a true value)
    or do not."""

    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone):
        """Constructor

        An option has a shortName which is a single character, a =
longName which is a word
        and a description.  For example if shortName =3D 'm' and =
longName =3D 'monty' then it
        will match command line options such as -m or --monty."""=0A=
        self.shortName, self.longName, self.desc =3D shortName, =
longName, desc=0A=
        self.value =3D None=0A=
=0A=
    def __str__(self): return str(self.value)=0A=
    def __nonzero__(self): return operator.truth(self.value)=0A=
=0A=
    def getoptShortFormat(self): return self.shortName=0A=
    def getoptLongFormat(self): return self.longName=0A=
=0A=
    def getOptionUsageHeader(self, indent, colWidth):=0A=
        "Generate descriptive header to use in help strings"
        tmp =3D []=0A=
        if self.shortName is not None: tmp.append('-' + self.shortName)=0A=
        if self.longName is not None: tmp.append('--' + self.longName)=0A=
        return string.join(tmp, ', ')=0A=
=0A=
    def getOptionUsageFooter(self, indent, colWidth):=0A=
        "Generate descriptive footer to use in help strings"
        return ''=0A=
=0A=
    def getOptionUsageBody(self, indent, colWidth):
        "Generate descriptive body to use in help strings"=0A=
        if self.desc is None: return ''=0A=
=0A=
        # we use a dumbwriter to reflow the paragraph for us=0A=
        s =3D cStringIO.StringIO()=0A=
        writer =3D formatter.DumbWriter(s, maxcol=3DcolWidth-indent)=0A=
        writer.send_flowing_data(self.desc)=0A=
        writer.flush()=0A=
=0A=
        # add margin to lines=0A=
        lines =3D string.split(s.getvalue(), '\n')=0A=
        lines =3D map(operator.add, [' '*indent] * len(lines), lines)=0A=
        return string.join(lines, '\n')=0A=
=0A=
    def getOptionUsage(self, indent=3D3, colWidth=3D70):
        """Generate the option description for help strings

        Uses getOptionUsageHeader, getOptionUsageFooter, =
getOptionUsageBody to
        build this text"""=0A=
        return string.join(filter(len, =
[self.getOptionUsageHeader(indent, colWidth),=0A=
                                        self.getOptionUsageBody(indent, =
colWidth),=0A=
                                        =
self.getOptionUsageFooter(indent, colWidth)]), '\n')=0A=
=0A=
    def processOption(self, opts, values):
        "Using the output of getopt, update this options state"=0A=
        try: posn =3D opts.index(self.shortName)=0A=
        except:=0A=
            try: posn =3D opts.index(self.longName)=0A=
            except: posn =3D None=0A=
        self.value =3D (posn is not None)=0A=
        return self.value=0A=
=0A=
BooleanOption =3D Option=0A=
=0A=
class HelpOption(Option):=0A=
    "An Option indicating a request for Help Information."=0A=
    def __init__(self, shortName=3D'h', longName=3D'help', =
desc=3D'Provides this message.'):=0A=
        Option.__init__(self, shortName, longName, desc)=0A=
=0A=
class ArgOption(Option):=0A=
    "An Option that expects an argument to follow."=0A=
    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone, =
default=3DNone, required=3D0):=0A=
        Option.__init__(self, shortName, longName, desc)=0A=
        self.default, self.required =3D default, required=0A=
=0A=
    def getoptShortFormat(self):=0A=
        if self.shortName: return self.shortName + ':'=0A=
        else: return None=0A=
=0A=
    def getoptLongFormat(self):=0A=
        if self.longName: return self.longName + '=3D'=0A=
        else: return None=0A=
=0A=
    def getOptionUsageHeader(self, indent, colWidth):=0A=
        tmp =3D []=0A=
        if self.shortName: tmp.append('-' + self.shortName + ' <value>')=0A=
        if self.longName: tmp.append('--' + self.longName + '=3D<value>')=0A=
        return string.join(tmp, ', ')=0A=
=0A=
    def getOptionUsageFooter(self, indent, colWidth):=0A=
        if self.default is not None:=0A=
            return ' ' * indent + 'Default is: ' + str(self.default)=0A=
        elif self.required:=0A=
            return ' ' * indent + 'This Option is Required.'=0A=
        else: return ''=0A=
=0A=
    def processOption(self, opts, values):=0A=
        try: posn =3D opts.index(self.shortName)=0A=
        except:=0A=
            try: posn =3D opts.index(self.longName)=0A=
            except: posn =3D None=0A=
=0A=
        if posn is None: self.value =3D self.default=0A=
        else: self.value =3D values[posn]=0A=
=0A=
        if self.value is None and self.required:=0A=
            raise OptionError, 'The option %s, %s is required.' % =
(self.shortName, self.longName)=0A=
=0A=
class _ConvertableOption(ArgOption):=0A=
    "An Option that applies a function to its input before returning it."=0A=
    def __init__(self, converterFn, shortName=3DNone, longName=3DNone, =
desc=3DNone, default=3DNone, required=3D0):=0A=
        """Constructor

        converterFn will be used to convert the text string parsed into =
a Python object."""
        ArgOption.__init__(self, shortName, longName, desc, default, =
required)=0A=
        self.converterFn =3D converterFn=0A=
=0A=
    def processOption(self, opts, values):=0A=
        ArgOption.processOption(self, opts, values)=0A=
        if self.value is not None and self.value !=3D self.default:=0A=
            try: self.value =3D self.converterFn(self.value)=0A=
            except StandardError:=0A=
                raise OptionError, \=0A=
                      'The option %s, %s with value %s cannot be =
translated to a %s.' % \=0A=
                      (self.shortName, self.longName, self.value, =
self.__class__.__name__)=0A=
=0A=
class EvalOption(_ConvertableOption):=0A=
    "An Option that uses eval to convert its text value."=0A=
    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone, =
default=3DNone, required=3D0):=0A=
        _ConvertableOption.__init__(self, eval, shortName, longName, =
desc, default, required)=0A=
=0A=
class ListOption(_ConvertableOption):=0A=
    """An Option that expects a list of comma-separated values.=0A=
=0A=
    The option value is a List."""=0A=
    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone, =
default=3DNone, required=3D0):=0A=
        _ConvertableOption.__init__(self, self.convert, shortName, =
longName, desc, default, required)=0A=
=0A=
    def convert(self, x):=0A=
        if x: return string.split(str(x), ',')=0A=
        else: return []=0A=
=0A=
class IntegerListOption(ListOption):=0A=
    """An Option that expects a list of comma-separated integers.=0A=
=0A=
    The option value is a List of integers."""=0A=
    def convert(self, x): return map(int, string.split(str(x), ','))=0A=
=0A=
class LongListOption(ListOption):=0A=
    """An Option that expects a list of comma-separated longs.=0A=
=0A=
    The option value is a List of integers."""=0A=
    def convert(self, x): return map(long, string.split(str(x), ','))=0A=
=0A=
class FloatListOption(ListOption):=0A=
    """An Option that expects a list of comma-separated floats.=0A=
=0A=
    The option value is a List of floats."""=0A=
    def convert(self, x): return map(float, string.split(str(x), ','))=0A=
=0A=
class ComplexListOption(ListOption):=0A=
    """An Option that expects a list of comma-separated complex numbers.=0A=
=0A=
    The option value is a List of complex numbers."""=0A=
    def convert(self, x): return map(complex, string.split(str(x), ','))=0A=
=0A=
class IntegerOption(_ConvertableOption):=0A=
    """An option that expects an integer argument.=0A=
=0A=
    The option value is converted to an integer value."""=0A=
    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone, =
default=3DNone, required=3D0):=0A=
        _ConvertableOption.__init__(self, int, shortName, longName, =
desc, default, required)=0A=
=0A=
class LongOption(_ConvertableOption):=0A=
    """An option that expects a long integer argument.=0A=
=0A=
    The option value is converted to a long integer value."""=0A=
    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone, =
default=3DNone, required=3D0):=0A=
        _ConvertableOption.__init__(self, long, shortName, longName, =
desc, default, required)=0A=
=0A=
class FloatOption(_ConvertableOption):=0A=
    """An option that expects a float argument.=0A=
=0A=
    The option value is converted to a float."""=0A=
    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone, =
default=3DNone, required=3D0):=0A=
        _ConvertableOption.__init__(self, float, shortName, longName, =
desc, default, required)=0A=
=0A=
class ComplexOption(_ConvertableOption):=0A=
    """An option that expects a complex argument.=0A=
=0A=
    The option value is a converted to a complex number."""=0A=
    def __init__(self, shortName=3DNone, longName=3DNone, desc=3DNone, =
default=3DNone, required=3D0):=0A=
        _ConvertableOption.__init__(self, complex, shortName, longName, =
desc, default, required)=0A=
=0A=
try:=0A=
    from mx import DateTime=0A=
    class DateOption(_ConvertableOption):=0A=
        """An option that expects a date argument.=0A=
=0A=
        The option value is a converted to an mxDateTime, using=0A=
        DateTime.Parser.DateFromString.  The safest format for dates=0A=
        wil be of the form '1 May 99'.  Note that DateFromString itself
        never fails, preferring instead to return you a dud date object.=0A=
=0A=
        This option is only supported on platforms with the mxDateTime =
package."""=0A=
        def __init__(self, shortName=3DNone, longName=3DNone, =
desc=3DNone, default=3DNone, required=3D0):=0A=
            _ConvertableOption.__init__(self, =
DateTime.Parser.DateFromString,=0A=
                                        shortName, longName, desc, =
default, required)=0A=
=0A=
    class DateListOption(ListOption):=0A=
        """An Option that expects a list of comma-separated dates.=0A=
=0A=
        The option values are converted to mxDateTimes, using=0A=
        DateTime.Parser.DateFromString.  The safest format for dates=0A=
        wil be of the form '1 May 99'. The option value is a List of =
mxDateTimes.=0A=
=0A=
        This option is only supported on platforms with the mxDateTime =
package."""=0A=
        def convert(self, x): return map(DateTime.Parser.DateFromString, =
string.split(str(x), ','))=0A=
=0A=
    class DateTimeOption(_ConvertableOption):=0A=
        """An option that expects a datetime argument.=0A=
=0A=
        The option value is a converted to an mxDateTime, using=0A=
        DateTime.Parser.DateTimeFromString.  The safest format for dates=0A=
        wil be of the form '1 May 99 16:15:30' (???).=0A=
=0A=
        This option is only supported on platforms with the mxDateTime =
package."""=0A=
        def __init__(self, shortName=3DNone, longName=3DNone, =
desc=3DNone, default=3DNone, required=3D0):=0A=
            _ConvertableOption.__init__(self, =
DateTime.Parser.DateTimeFromString,=0A=
                                        shortName, longName, desc, =
default, required)=0A=
=0A=
except ImportError: pass=0A=
=0A=
class Parser:
    """An option parser

    The Parser is constructed with a list of options that it will =
understand, and
    optionally header text to use in any help dumps.

    After construction the user should call parse().  This will return a =
list
    representing the remainder of the line unparsed.  After parse()'ing, =
the option
    objects the parser controls will have changed their state.  Their =
values will be
    available ion their .value attribute."""=0A=
    def __init__(self, options=3D[], headerText=3D''):=0A=
        self.options =3D list(options)=0A=
        self.headerText =3D str(headerText)=0A=
        if len(self.headerText) and self.headerText[-1] !=3D '\n':=0A=
            self.headerText =3D self.headerText + '\n'=0A=
=0A=
        self._addHelpOption()=0A=
=0A=
    def _addHelpOption(self):=0A=
        "If a help option is missing, attempts to construct one"=0A=
        if not self.findHelpOption():=0A=
            helpShortName =3D ('h' not in map(lambda x: x.shortName, =
self.options) and 'h') or \=0A=
                            ('?' not in map(lambda x: x.shortName, =
self.options) and '?') or None=0A=
            helpLongName =3D ('help' not in map(lambda x: x.longName, =
self.options) and 'help') or None=0A=
=0A=
            if helpShortName or helpLongName:=0A=
                optHelp =3D HelpOption(helpShortName, helpLongName)=0A=
                self.options.append(optHelp)=0A=
=0A=
    def parseArguments(self, arguments=3DNone):
        "Parse the arguments, updating the state of any controlled =
option objects."=0A=
        if arguments is None: arguments =3D sys.argv=0A=
=0A=
        # build up the option strings=0A=
        shortArgs =3D string.join(\=0A=
            filter(lambda x: x, map(lambda x: x.getoptShortFormat(), =
self.options)), '')=0A=
        longArgs =3D filter(lambda x: x, map(lambda x: =
x.getoptLongFormat(), self.options))=0A=
=0A=
        # use getopt to parse the command line=0A=
        try: optPairs, self.dregs =3D getopt.getopt(arguments[1:], =
shortArgs, longArgs)=0A=
        except getopt.error, e:=0A=
            print e=0A=
            print self.provideHelpString()=0A=
            raise ParserError, 'Arguments unparsed as option errors =
occurred'=0A=
=0A=
        # split up into options and their values if any=0A=
        opts, values =3D map(operator.getitem, optPairs, =
[0]*len(optPairs)), \=0A=
                       map(operator.getitem, optPairs, [1]*len(optPairs))=0A=
        opts =3D map(lambda x: string.replace(x, '-', ''), opts)=0A=
=0A=
        # first check if theres a help option=0A=
        hOpt =3D self.findHelpOption()=0A=
        if hOpt is not None:=0A=
            hOpt.processOption(opts, values)    # process me=0A=
            if hOpt:=0A=
                print self.provideHelpString()=0A=
                raise HelpError, 'Arguments were unparsed, help was =
requested.'=0A=
=0A=
        # process options, trapping errors as they occur and dumping =
details=0A=
        errList =3D []=0A=
        for o in self.options:=0A=
            try: o.processOption(opts, values)=0A=
            except OptionError, e:=0A=
                errList.append(e)=0A=
=0A=
        # if at least one error occurred, dump help to stdout=0A=
        if len(errList):=0A=
            print 'Option Parsing Errors'=0A=
            print '---------------------'=0A=
            for e in errList: print '%s: %s' % (e.__class__.__name__, e)=0A=
            print '---------------------'=0A=
            print self.provideHelpString()=0A=
            raise ParserError, 'Arguments unparsed as option errors =
occurred'=0A=
=0A=
        return self.options, self.dregs=0A=
=0A=
    def parse(self, arguments=3DNone):
        "Parse the arguments, providing help on failure"=0A=
        try: opts, remainder =3D self.parseArguments()=0A=
        except HelpError: sys.exit(0)=0A=
        except OptionError: sys.exit('Terminated due to Option =
Processing Error')=0A=
=0A=
        return remainder=0A=
=0A=
    def findHelpOption(self):
        "Helper method, find any help options"=0A=
        hOpts =3D filter(lambda x: issubclass(x.__class__, HelpOption), =
self.options)=0A=
        if len(hOpts): return hOpts[0]=0A=
        else: return None=0A=
=0A=
    def provideHelpString(self, indent=3D3, colWidth=3D70):=0A=
        "Return a help string combining all of the options information"
        return self.headerText + 'The following options are =
recognised:\n' + \=0A=
               string.join(map(lambda x,i=3Dindent,w=3DcolWidth: =
x.getOptionUsage(i,w), self.options), '\n') + '\n\n' + \=0A=
               '(Note: The argument -- terminates option processing, =
remaining arguments will be unparsed.)'=0A=
=0A=
=0A=
    def exit(self, prefix=3D'', suffix=3D''):
        "Dump help and exit the program."=0A=
        sys.exit(prefix + self.provideHelpString() + suffix)=0A=
=0A=
if __name__ =3D=3D '__main__':=0A=
    # some options to test out, we declare them as shortName, longName, =
description, ...
    optInt =3D IntegerOption('i', 'integer', 'An Integer', default=3D4)
    optList =3D ListOption(None, 'list', 'A string list')
    optList2 =3D ListOption('l', 'list2', 'A required string list', =
required=3D1)
    optEval =3D EvalOption('e', 'eval', 'Evaluated expression')
    optBoolean =3D Option('b', 'boolean', 'An on/off option')
    optFloatList =3D FloatListOption('f', 'floats', 'A listof floats', =
default=3D[2.0, 4.1])
    optString =3D ArgOption('s', 'string', 'A string', required=3D1)
   =20
    # construct the parser    =0A=
    p =3D Parser([optInt, optList, optList2, optEval, optBoolean, =
optFloatList, optString],
               "You're testing the option parser !!")

    # call the parser.  This will return the remainder of the command =
line.
    # lets pretend we dont expect anything and so die if there is =
something
    p.parse() and p.exit('There should be no remaining command line =
arguments\n\n')

    # Now you can access the arguments as eg optInt.value
    print "Here are the option values after the parse"
    for opt in p.options:
        print opt.getOptionUsageHeader(None, None), "has value", =
opt.value, "of type", type(opt.value)
       =20
    
------=_NextPart_000_0073_01C1B42C.FC76F660--



From derek@chocolate-fish.com  Wed Feb 13 01:42:03 2002
From: derek@chocolate-fish.com (Derek Harland)
Date: Wed, 13 Feb 2002 01:42:03 -0000
Subject: [getopt-sig] Re: Prefer non-imperative -- my 1 cent + Change
Message-ID: <007e01c1b42f$a42475c0$080a0a0a@g7n3g0>

Russ Cox rsc@plan9.bell-labs.com wrote
> Can you give an example (perhaps a big one) where option
> parsing with Optik-style objects makes things clearer
> than the traditional loop?

In a loop framework it often seems that flags end up being kept to decide
what to do when another option is met ... perhaps your action when you see B
depends on if A was true ... maybe you need to defer action until you see
both.

In the object orientated world that may look easier.  Well, *I* think it
looks easier.
Ok, well I dont know Optik so I'll answer this from the point of view of my
option parsing module (named, inspiringly, option and posted earlier, sorry
no web source currently).  I expect Optik to be roughly similar

Assuming they are just straight boolean options you would go say

optA, optB = Option('a'), Option('b')
Parser([optA, optB]).parse()

if optA and optB: functionForA_and_B()
elif optA: onlyA()
else: onlyB()

In a loop wouldnt you have to defer your choice until all the arguments are
parsed ... which makes the loop just a "oh its an A, lets set an A flag
somewhere" ?

> All that aside, my big problem with the Optik-style
> modules, and to a lesser extent getopt, is the barrier

Frankly, I find getopt itself quite opaque.

> to entry.  Option parsing is something everyone has to
> do, more or less; it should require as little work
> as possible to get simple parsing in place.  The fact
> that it's _not_ simple enough is why people settle for
> quick and dirty things like
>
>  if sys.argv[1]=='-d':
>   debug=1
>   sys.argv = sys.argv[1:]
>
> when they want just one or two options.  This problem
> is certainly endemic in C programs; I haven't surveyed
> enough Python programs to make informed comments about
> how widespread the practice is in Python.

Its widespread ;-)

> On the other hand, since (in the module I posted)
> it's hardly any more code to use the option parser:
>
>  opt = OptionParser()
>  for o in opt:
>   if o=='-d':
>    debug=1

Again from the point of view of my option module

optDebug = Option('d', 'debug')
Parser([optDebug]).parse()

Thats two lines not four.  Plus you get automatic help options installed.
Plus if they pass you a -c it will complain and say its an invalid option
and dump help.  Plus if you want to know if debug has been set then you can
just test it ... eg

if optDebug: print "Debug info here"

Is it as easy in terms of entry level? No, but I think thats solely because
of the name of Parser() which is a scary term.

> Of course, it's possible that there is some hybrid scheme
> that would satisfy everyone, but I really don't have
> any feel for argument parsing so hairy that you need
> parsing objects and callbacks and the like: someone please
> enlighten me.

I'm not sure if this brings you any closer to Buddha ?

Des.




From pac1@tiac.net  Wed Feb 13 01:29:28 2002
From: pac1@tiac.net (Patrick Callahan)
Date: Tue, 12 Feb 2002 20:29:28 -0500
Subject: [getopt-sig] Re: [optik] ANNOUNCE: getopt-sig created
In-Reply-To: <20020212195838.GA27186@gerg.ca>
References: <20020212195838.GA27186@gerg.ca>
Message-ID: <200202130129.g1D1TIPP030996@relayhost.us.inter.net>

Its good to see that this is going to get some discussion.  I think Greg 
Ward's optik implementation is quite clear and focused.  I've not used the 
other two yet.

I'm wondering a first step would be to develop a list of what differentiates 
optik, arglist.py and Russ Cox's Plan9 style implementation.

I hope the principal authors of the (so far) three competing packages can put 
their heads together and come up with a merger of the best features of all 
three for inclusion in Python 2.3  If it turns out there's more than one 
distinct set of functionality that needs to be defined under the general 
heading of "option parsing".  I think that would be ok too.

- Pat Callahan





From s.keim  Wed Feb 13 11:55:21 2002
From: s.keim (s.keim)
Date: Wed, 13 Feb 2002 12:55:21 +0100
Subject: [getopt-sig] some comments about Optick
Message-ID: <8EE23156-2078-11D6-B507-0050E4A06334@laposte.net>

After a first dive into Optick documentation, I think that it seems very 
promising.

I have some questions/suggestions about the API:

  * I don't see the need for the level of indirection in the add_option 
command.
Wouldn't it be easier for the user to directly use functions objects:
for sample:
parser.add_option("-f", "--file", type=string, dest="filename",
                         help="read data from FILENAME")
or
parser.add_option("-v", "--verbose",  action=parser.store_true,  
dest="verbose")


* It would be better to define call-back as functions with one arg. This 
will allow you to change/extend the API only by appending attributes to 
the argument and then without breaking existing call-back functions.

* Parsing value policy
in the doc:
 >Let's parse another fake command-line.  This time, we'll jam the option
 >argument right up against the option -- "-n42" (one argument) is
 >equivalent to "-n 42" (two arguments).
in the second case, how the parser know that 42 is the value of the -n 
argument and not  a positional parameter? Is it because we have defined 
that action="store"?

* This is mainly a matter of taste, but I'd like to have the dest 
argument optional, if the long option is defined:
parser.add_option("-v", "--verbose",  action="store_true")
would be equivalent to:
parser.add_option("-v", "--verbose",  action="store_true",  
dest="verbose")

* required parameters
I haven't see this in documentation, so I suggest to add a field to 
options:
Option("-f", action="store", type="string", dest="filename", required=1)
Then parser will fail if the -f option doesn't exist in the command line

*groups
Sometime, you want to manage options by groups:
- the user can only supply one of the options of the group.
- the user must supply at least one option of the group.
I don't know how to express this yet, but this would be something 
interesting to have. In fact this is probably quite related to the 
"store_const" action.

* default values
I think that the command line parser shouldn't manage default values 
because quite often software provide several way to define options:
- hard coded
- site dependent configuration file
- user configuration file
- environ variables
- registry base on windows
- command line

So if default value management was left to a separate object, we could 
have something like:
file_parser  = FileOptionParser(xxxx)
env_parser = EnvironParser( xxxx )
cmd_parser = OptionParser(option_list= xxxx)

options = DefaultValues (pos1, "positional2", verbose=1, 
filename="'/tmp/usr.tmp' )
options.update(file_parser.parse('/usr/local/share/prog/defaults'))
options.update(file_parser.parse('~/.prog/defaults'))
options.update(env_parser.parse())
options.update(cmd_parser.parse(sys.argv[1:] ))

and then go to:
if options.verbose:
           print "reading %s..." % options.filename

By the way, many informations are shared between the xxxx arguments of 
the several parsers, so maybe we could look for an api that would 
permit  to avoid duplication.

seb



From rsc@plan9.bell-labs.com  Wed Feb 13 18:24:50 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Wed, 13 Feb 2002 13:24:50 -0500
Subject: [getopt-sig] there isn't really any discussion here
Message-ID: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com>

What I've learned from getopt-sig is that the world appears to be
divided in two.

In one camp, there are people who believe that the argument parser
should be helping to manage the complexity of the program, believe
that the current getopt does too little, believe that objects are the
solution to the problem, believe the argument parser should include a
grab bag of helper functions to do everything you could imagine
wanting, and aren't bothered by the complexity of the whole approach.
This camp seems quite large, or at least vocal.  Perhaps that's in
part because the creation of getopt-sig was announced on optik-users,
but not entirely: there are so many object-based argument parsers out
there that there must also be plenty of people who believe this is the
way to go.

In the other camp, there are people who believe that the argument
parser should be parsing arguments and doing little else, believe that
the current getopt does too much (mainly because it requires
specifying the argument set twice), and believe that writing a for
loop over the arguments is convenient and even preferable.  This camp
seems quite small, or at least quiet.  The only people who have spoken
who might fall into this camp besides me are Paul Prescod and
Martin v.  Loewis, neither of whom has joined getopt-sig.

It's not clear that one group is right and one group is wrong.  I
think it's basically a difference of opinion about what constitutes
the best approach to the problem.  Unfortunately I don't see any good
way to sate both groups simultaneously.  I am ready to admit defeat.
Perhaps it is worth having both approaches in the standard library, as
IterGetopt and ObjectGetopt, or something like that.  Perhaps not.

I admit to being somewhat disheartened by the recurring cries for
supposed features like ``optional arguments'' (which introduce parsing
ambiguities) and ``mandatory options'' (a contradiction in terms).

I did gain something from all this, namely that I now have an
argument parser that is simpler to use than the current getopt.
Now I won't need to consult the getopt documentation every
time I write a main() function.  For that I am quite grateful.

Russ



From gward@python.net  Wed Feb 13 20:52:04 2002
From: gward@python.net (Greg Ward)
Date: Wed, 13 Feb 2002 15:52:04 -0500
Subject: [getopt-sig] some comments about Optick
In-Reply-To: <8EE23156-2078-11D6-B507-0050E4A06334@laposte.net>
References: <8EE23156-2078-11D6-B507-0050E4A06334@laposte.net>
Message-ID: <20020213205204.GF2360@gerg.ca>

On 13 February 2002, s. keim said:
> After a first dive into Optick documentation, I think that it seems very 
> promising.
> 
> I have some questions/suggestions about the API:

I'm not sure if this discussion belongs here or on the optik-users list.
This list should be for comparing/contrasting the various getopt
replacements/enhancements.  Oh well, here I go.


>  * I don't see the need for the level of indirection in the add_option 
> command.
> Wouldn't it be easier for the user to directly use functions objects:
> for sample:
> parser.add_option("-f", "--file", type=string, dest="filename",
>                         help="read data from FILENAME")

That presupposes that you have imported the name 'string' from
somewhere; since this conflicts with a standard library module, I'm not
too keen on it.

Or did you mean to put a type object there, like StringType or 'str'
(same thing in Python 2.2)?  Two problems with that:

  * tricky to make it work the same way < 2.2 and >= 2.2:
    if you want either 'str' or StringType to work in all versions
    of Python (seems desirable to me), then you have a work a
    little bit harder

  * what about extensibility?  one of the major features of Optik's
    type system is that its extensible: you could add a "file" or 
    "dir" or "outfile" type; some of those things correspond to
    Python types or builtins, and some don't

> or
> parser.add_option("-v", "--verbose",  action=parser.store_true,  
> dest="verbose")

...and that presupposes a name 'store_true' in each OptionParser
instance, which means that adding an action requires subclassing both
Option and OptionParser.

Using strings to represent types and actions might be slightly more
verbose, but I think it makes it easier to extend Optik: all you need to
do is recognize one more string.

> * It would be better to define call-back as functions with one arg. This 
> will allow you to change/extend the API only by appending attributes to 
> the argument and then without breaking existing call-back functions.

I took the opposite approach: go ahead and pass everything that might
someday be useful to some callbacks, so I don't have to extend the API
in future.  ;-) (Famous last words...)

> * Parsing value policy
> in the doc:
> >Let's parse another fake command-line.  This time, we'll jam the option
> >argument right up against the option -- "-n42" (one argument) is
> >equivalent to "-n 42" (two arguments).
> in the second case, how the parser know that 42 is the value of the -n 
> argument and not  a positional parameter? Is it because we have defined 
> that action="store"?

No, it's because this option has a type and therefore takes a value.
(action="store" without a type is (currently) an error; I'm thinking of
making the default type "string" because that's the usual case.)

> * This is mainly a matter of taste, but I'd like to have the dest 
> argument optional, if the long option is defined:
> parser.add_option("-v", "--verbose",  action="store_true")
> would be equivalent to:
> parser.add_option("-v", "--verbose",  action="store_true",  
> dest="verbose")

Ooh, good idea!  Filed in my mental to-do list.

> * required parameters
> I haven't see this in documentation, so I suggest to add a field to 
> options:
> Option("-f", action="store", type="string", dest="filename", required=1)
> Then parser will fail if the -f option doesn't exist in the command line

The phrase "required option" is an oxymoron.  Think about it.  (Also,
look in the optik-users archive -- this was discussed a month or so
ago.)

(But if enough users want the damn feature, I'll probably add it.  I'm
like that.)

> *groups
> Sometime, you want to manage options by groups:
> - the user can only supply one of the options of the group.
> - the user must supply at least one option of the group.
> I don't know how to express this yet, but this would be something 
> interesting to have. In fact this is probably quite related to the 
> "store_const" action.

No.  Optik knows anything about inter-option dependencies, nor should
it.  You can write your own code to do that after parser.parse_args()
returns.

> * default values
> I think that the command line parser shouldn't manage default values 
> because quite often software provide several way to define options:

So don't use that feature.  Quite often software takes options from the
command-line and nowhere else, and then having default values is really
handy.

> By the way, many informations are shared between the xxxx arguments of 
> the several parsers, so maybe we could look for an api that would 
> permit  to avoid duplication.

That's kind of the point of this sig: get something better than getopt
in the standard library, that everyone can live with.

        Greg
-- 
Greg Ward - Unix nerd                                   gward@python.net
http://starship.python.net/~gward/
Don't hate yourself in the morning -- sleep till noon.


From gward@python.net  Wed Feb 13 20:57:02 2002
From: gward@python.net (Greg Ward)
Date: Wed, 13 Feb 2002 15:57:02 -0500
Subject: [getopt-sig] there isn't really any discussion here
In-Reply-To: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com>
References: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com>
Message-ID: <20020213205702.GG2360@gerg.ca>

On 13 February 2002, Russ Cox said:
> What I've learned from getopt-sig is that the world appears to be
> divided in two.

Yep, that's pretty clear.  About the only thing we disagree on is
personal preference.

> It's not clear that one group is right and one group is wrong.  I
> think it's basically a difference of opinion about what constitutes
> the best approach to the problem.  Unfortunately I don't see any good
> way to sate both groups simultaneously.  I am ready to admit defeat.
> Perhaps it is worth having both approaches in the standard library, as
> IterGetopt and ObjectGetopt, or something like that.  Perhaps not.

I've been thinking along those lines myself.  I think I'd call them
OptionParser and OptionIterator, but never mind.  I plan to do some
experimental hacking-up of the Optik code to see if a simple iterator
interface can be ripped out of OptionParser, with all the bondage and
discipline that I like dropped on the floor.  It might just be possible
to please everyone.

> I admit to being somewhat disheartened by the recurring cries for
> supposed features like ``optional arguments'' (which introduce parsing
> ambiguities) and ``mandatory options'' (a contradiction in terms).

I mostly agree.  I admit to wanting "optional arguments" for some
applications, but I fear the ambiguity.  I totally agree about "required
options".

> I did gain something from all this, namely that I now have an
> argument parser that is simpler to use than the current getopt.
> Now I won't need to consult the getopt documentation every
> time I write a main() function.  For that I am quite grateful.

Cool!  Can you put your code on the web somewhere and post a URL?

        Greg
-- 
Greg Ward - geek-at-large                               gward@python.net
http://starship.python.net/~gward/
I'm on a strict vegetarian diet -- I only eat vegetarians.


From gward@python.net  Wed Feb 13 21:03:37 2002
From: gward@python.net (Greg Ward)
Date: Wed, 13 Feb 2002 16:03:37 -0500
Subject: [getopt-sig] some comments about Optick
In-Reply-To: <20020213205204.GF2360@gerg.ca>
References: <8EE23156-2078-11D6-B507-0050E4A06334@laposte.net> <20020213205204.GF2360@gerg.ca>
Message-ID: <20020213210337.GA2522@gerg.ca>

On 13 February 2002, I said:
> No.  Optik knows anything about inter-option dependencies, nor should
> it.  You can write your own code to do that after parser.parse_args()
> returns.

Typo: Optik *doesn't* know anything about inter-option dependencies.

        Greg
-- 
Greg Ward - just another /P(erl|ython)/ hacker          gward@python.net
http://starship.python.net/~gward/
"I know a lot about art, but I don't know what I like!"


From derek@chocolate-fish.com  Wed Feb 13 20:58:17 2002
From: derek@chocolate-fish.com (Derek Harland)
Date: Wed, 13 Feb 2002 20:58:17 -0000
Subject: [getopt-sig] there isn't really any discussion here
References: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com>
Message-ID: <004001c1b4d2$acec8c80$080a0a0a@g7n3g0>

----- Original Message -----
From: "Russ Cox" <rsc@plan9.bell-labs.com>
To: <getopt-sig@python.org>
Sent: Wednesday, February 13, 2002 6:24 PM
Subject: [getopt-sig] there isn't really any discussion here


> I admit to being somewhat disheartened by the recurring cries for
> supposed features like ``optional arguments'' (which introduce parsing
> ambiguities) and ``mandatory options'' (a contradiction in terms).

I'm not sure I understand either of these worries.  To me an optional
argument is usually a simple flag to the program to indicate something.  I
mean you either compile a c program in debug mode, or you  don't.
Alternatively its something you specify if required, otherwise the program
decides it for you.

Then again some other options are mandatory but there values are not known
in advance.

eg.  I have scripts that do something and need to know the date for which
they are processing data.  You specify it with --close=20020201 eg if you
wish to run it historically.  Otherwise it assumes you mean today (as you
nearly always do) ... thats an optional argument to me.

This same script only processes data relating to a specific currency.  You
have to give a currency for it to operate on --currency=EUR.  It can't run
without knowing a currency to operate on.  Thats mandatory.

Object orientated, iterative, getopt or purely inspecting sys.argv ... lots
of scripts *have* to be given something.

> I did gain something from all this, namely that I now have an
> argument parser that is simpler to use than the current getopt.
> Now I won't need to consult the getopt documentation every
> time I write a main() function.  For that I am quite grateful.

We may disagree about the approach, but this sums up the problem entirely.
I dislike getopts in all of its incarnations ... remembering that colons
represent argument options, that the return values include the '-' or '--'
in the option name.  It just isn't very friendly, which is Python's
underlying strength.

But without *something* better people will always be forced to reinvent an
obviously square wheel.

> Russ

Des



From rsc@plan9.bell-labs.com  Wed Feb 13 21:21:27 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Wed, 13 Feb 2002 16:21:27 -0500
Subject: [getopt-sig] there isn't really any discussion here
Message-ID: <5a6bbdb8b2d5e9950b54b2a17f5946ef@plan9.bell-labs.com>

> > I admit to being somewhat disheartened by the recurring cries for
> > supposed features like ``optional arguments'' (which introduce parsing
> > ambiguities) and ``mandatory options'' (a contradiction in terms).
> 
> I'm not sure I understand either of these worries.  To me an optional
> argument is usually a simple flag to the program to indicate something.  I
> mean you either compile a c program in debug mode, or you  don't.
> Alternatively its something you specify if required, otherwise the program
> decides it for you.

That's not what I mean by ``optional arguments''.  What I mean is some people
want to define a flag -c that might or might not take an argument.
In this case, it introduces an ambiguity.  If -c takes an optional argument,
does "foo -cd" mean "foo -c -d" or "foo -c d".  What if I want to specify
-c without an argument but then have a normal command-line argument
follow?  If you've got long options with their '--option=argument' syntax,
there's no ambiguity to making the '=argument' not optional.

> This same script only processes data relating to a specific currency.  You
> have to give a currency for it to operate on --currency=EUR.  It can't run
> without knowing a currency to operate on.  Thats mandatory.

Then I don't see why it's not a command line argument.  Why not just 
define that the first thing on the command line after the options
is the currency?  By your scheme, I would have to say 
"ssh --host=mycomputer" instead of "ssh mycomputer".
I don't see the point of using option syntax when the option
isn't really optional.

Russ



From gward@python.net  Wed Feb 13 21:23:46 2002
From: gward@python.net (Greg Ward)
Date: Wed, 13 Feb 2002 16:23:46 -0500
Subject: [getopt-sig] there isn't really any discussion here
In-Reply-To: <004001c1b4d2$acec8c80$080a0a0a@g7n3g0>
References: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com> <004001c1b4d2$acec8c80$080a0a0a@g7n3g0>
Message-ID: <20020213212346.GA2555@gerg.ca>

On 13 February 2002, Derek Harland said:
> I'm not sure I understand either of these worries.  To me an optional
> argument is usually a simple flag to the program to indicate something.  I
> mean you either compile a c program in debug mode, or you  don't.
> Alternatively its something you specify if required, otherwise the program
> decides it for you.

Let's get some terminology straight.  Howl if you think these
definitions are bogus:

  argument
    something in sys.argv (or possibly sys.argv[1:], depending on
    your PoV)

  option
    an argument used to supply extra information to guide or
    customize the execution of a program.  In Unix, options usually
    start with "-" or "--"; in DOS/VMS with "/".

  option argument
    an argument that follows an option and which is closely associated
    with that option, and consumed from the argument list when the
    option is.  Often, option arguments may also be included in the same
    argument as the option, eg.
      ["-f", "foo"]
    may be equivalent to
      ["-ffoo"]

    Some options never take an argument.  Some options always take an
    argument.  Lots of people want an "optional option arguments"
    feature, meaning that some options will take an argument if they see
    it, and won't if they don't.  This is somewhat controversial.

  positional argument
    something leftover in sys.argv after options have been parsed, ie.
    options and option arguments removed from the argument list.
    
    (Note that Optik does not change sys.argv, but returns a copy of
    it with only positional args left.  Other getopt replacements
    might work like this, or they might not.)

  required option
    an option that must be supplied on the command-line; this is
    an oxymoron and I personally consider it poor UI design.
    (I gather Russ Cox agrees with me.)  Any getopt replacement should
    make it fairly easy to implement required options, though, because
    lots of people want them.

        Greg
-- 
Greg Ward - nerd                                        gward@python.net
http://starship.python.net/~gward/
There's nothing to see here.  Move on; go about your business.


From rsc@plan9.bell-labs.com  Wed Feb 13 21:32:23 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Wed, 13 Feb 2002 16:32:23 -0500
Subject: [getopt-sig] my current argument parser
Message-ID: <0dfed672c11c9ac0990608e89b59f041@plan9.bell-labs.com>

[The docstring at the beginning explains all.  Comments welcome.]

from __future__ import generators
import sys, copy

"""Simple command-line argument iterator.

A typical example looks like:

	arg = ArgParser('usage: example [-d] [-r root] database\n')
	for o in arg:
		if o=='-d':
			debug=1
		elif o=='-r':
			root = arg.nextarg()
		else:
			arg.error('unknown option '+o)
	if len(arg.argv) != 1:
		arg.error()

This illustrates all the important points:

	* arg is a generator that returns the next argument.
	  >>>The behavior of the generator depends on the behavior
	  of the body of your for loop.<<<  This is why you don't have
	  to tell the ArgParser about your options a priori.  If an option
	  has an argument, you call arg.nextarg() to fetch it.  (If you
	  want an option to have two arguments, call arg.nextarg twice.)

	 * After the loop has finished, arg.argv contains the 
	  remaining command-line arguments.

	 * Calling arg.error prints the optional passed string,
	  then prints the usage message, then exits.

A few other points:

	* Consistent with typical Unix conventions, option parsing ends
	  after seeing '--', before seeing '-', or before any command-line argument
	  not beginning with a - (that wasn't gobbled as an option argument).
	  Assuming that -c does not take an argument but -f does, the following
	  set of command lines and parsings illustrates the rules:
		foo -c -- -a
			opts: -c
			args: -a
		foo -c - -a
			opts: -c
			args: - -a
		foo -c bar baz -a
			opts: -c
			args: bar baz -a
		foo -cfbar baz -a
			opts: -c -f
				(-f took bar)
			args: baz -a
		foo -cf bar baz -a
			opts: -c -f
				(-f took bar)
			args: baz -a
		foo -fc bar baz -a
			opts: -f
				(-f took c)
			args: bar baz -a

	* Single character short options begin with - and take arguments from the
	  rest of the current argument or from the next argument.  For example,
	  	foo -cbar
	  	foo -c bar
	  are equivalent.  There is no support for optional arguments, since that
	  introduces command-line parsing ambiguities.

	* Long options begin with -- and take arguments from an optional
	  ``=ARG'' suffix.  For example,
	  	foo --long=bar
	  is the only way to specify an argument to the --long option.
	  If the handler for '--long' does not fetch the argument with arg.nextarg,
	  the parser will call arg.error automatically.  There is support for optional
	  arguments to long options, since that does not introduce any ambiguities.
	  To fetch an optional argument call arg.nextarg(allownone=1).  If there
	  is no argument, None will be returned.

"""

class ArgError(Exception):
	def __init__(self, msg=None):
		if msg:
			self.msg = msg
		
class ArgParser:
	def __init__(self, usage, argv=sys.argv):
		self.argv0 = argv[0]
		self.argv = argv[1:]
		self.usage = usage
		self.waitingarg = ''

	def __iter__(self):
		# this assumes the "
		while self.argv:
			if self.argv[0]=='-' or self.argv[0][0]!='-':
				break
			a = self.argv.pop(0)
			if a=='--':
				break
			if a[0:2]=='--':
				i = a.find('=')
				if i==-1:
					self.waitingarg=None
					self.option = a
					yield self.option
					self.option = None
				else:
					self.waitingarg = a[i+1:]
					self.option = a[0:i]
					yield self.option
					if self.waitingarg:		# wasn't fetched using optarg
						self.error(self.option+' does not take an argument')
					self.option = None
				continue
			self.waitingarg = a[1:]
			while self.waitingarg:
				a = self.waitingarg[0:1]
				self.waitingarg = self.waitingarg[1:]
				self.option = '-'+a
				yield self.option
				self.option = None

	def nextarg(self, allownone=0):
		if self.waitingarg==None:
			if allownone:
				return None
			self.error(self.option+' requires an argument')
		elif self.waitingarg:
			ret = self.waitingarg
			self.waitingarg=''
		else:
			try:
				ret = self.argv.pop(0)
			except IndexError:
				self.error(self.option+' requires an argument')
		return ret

	def error(self, msg=None):
		if msg:
			sys.stderr.write('argument error: '+msg+'\n')
		sys.stderr.write(self.usage)
		sys.stderr.flush()
		sys.exit(1)



From gward@python.net  Wed Feb 13 23:10:15 2002
From: gward@python.net (Greg Ward)
Date: Wed, 13 Feb 2002 18:10:15 -0500
Subject: [getopt-sig] there isn't really any discussion here
In-Reply-To: <Pine.GSO.4.21.0202131700480.10936-100000@harper.uchicago.edu>
References: <20020213212346.GA2555@gerg.ca> <Pine.GSO.4.21.0202131700480.10936-100000@harper.uchicago.edu>
Message-ID: <20020213231015.GA2972@gerg.ca>

On Wed, 13 Feb 2002, I wrote:
>Let's get some terminology straight.  Howl if you think these
>definitions are bogus:
>
>  argument
>    something in sys.argv (or possibly sys.argv[1:], depending on
>    your PoV)

On 13 February 2002, Ben Wolfson replied (in private email; I
presume this was a mistake, as this really belongs on the list):
> I would prefer it if an option parser did *not* assume the arguments were
> in sys.argv.

First, I don't think that my off-the-cuff definitions should be
interpreted as carved-in-stone specifications for a future (or present)
getopt replacement.  But, just to be extra-super-careful, I'll revise
that one:

  argument
    one of the elements of sys.argv[1:], or a similar list provided
    to substitute for sys.argv[1:]


> I prefer doing something like this:
> 
> def main(av):
>     #parse options in av and do whatever the script does
> 
> if __name__ == '__main__':
>    main(sys.argv) #or sys.argv[1:]

I think that should be *allowed*, but I also don't like having to
explicitly refer to sys.argv[1:] in every single script I write.  That
explains Optik's interface:

        parse_args(args : [string] = sys.argv[1:],
                   values : Values = None)
        -> (values : Values, args : [string])

...ie. you can pass in an arg list if you want; if you don't, it uses
sys.argv[1:].  This is *so* obvious to me; I can't understand why other
getopt libraries (in Python at least, where sys.argv is globally
available) aren't like this.

> That way it's possible to import a script and call its main() with your own
> psuedo-command-line arguments.

Key word: *possible*, not required.

        Greg
-- 
Greg Ward - programmer-at-big                           gward@python.net
http://starship.python.net/~gward/
If at first you don't succeed, redefine success.


From wolfson@midway.uchicago.edu  Wed Feb 13 23:19:20 2002
From: wolfson@midway.uchicago.edu (Ben Wolfson)
Date: Wed, 13 Feb 2002 17:19:20 -0600 (CST)
Subject: [getopt-sig] there isn't really any discussion here
In-Reply-To: <20020213231015.GA2972@gerg.ca>
Message-ID: <Pine.GSO.4.21.0202131719170.22329-100000@harper.uchicago.edu>

On Wed, 13 Feb 2002, Greg Ward wrote:

>On Wed, 13 Feb 2002, I wrote:
>>Let's get some terminology straight.  Howl if you think these
>>definitions are bogus:
>>
>>  argument
>>    something in sys.argv (or possibly sys.argv[1:], depending on
>>    your PoV)
>
>On 13 February 2002, Ben Wolfson replied (in private email; I
>presume this was a mistake, as this really belongs on the list):
>> I would prefer it if an option parser did *not* assume the arguments were
>> in sys.argv.
>
>First, I don't think that my off-the-cuff definitions should be
>interpreted as carved-in-stone specifications for a future (or present)
>getopt replacement.  But, just to be extra-super-careful, I'll revise
>that one:
>
>  argument
>    one of the elements of sys.argv[1:], or a similar list provided
>    to substitute for sys.argv[1:]
>
>
>> I prefer doing something like this:
>> 
>> def main(av):
>>     #parse options in av and do whatever the script does
>> 
>> if __name__ == '__main__':
>>    main(sys.argv) #or sys.argv[1:]
>
>I think that should be *allowed*, but I also don't like having to
>explicitly refer to sys.argv[1:] in every single script I write.  That
>explains Optik's interface:
>
>        parse_args(args : [string] = sys.argv[1:],
>                   values : Values = None)
>        -> (values : Values, args : [string])
>
>...ie. you can pass in an arg list if you want; if you don't, it uses
>sys.argv[1:].  This is *so* obvious to me; I can't understand why other
>getopt libraries (in Python at least, where sys.argv is globally
>available) aren't like this.
>
>> That way it's possible to import a script and call its main() with your own
>> psuedo-command-line arguments.
>
>Key word: *possible*, not required.
>
>        Greg
>

-- 
BTR    
             Mean, as in:  Ben's Risotto: I'LL CUT YOU!
                      Or:  Ben's Risotto: Nobody likes you.
                              -- Steve Christensen



From derek@chocolate-fish.com  Wed Feb 13 23:25:23 2002
From: derek@chocolate-fish.com (Derek Harland)
Date: Wed, 13 Feb 2002 23:25:23 -0000
Subject: [getopt-sig] there isn't really any discussion here
References: <5a6bbdb8b2d5e9950b54b2a17f5946ef@plan9.bell-labs.com>
Message-ID: <009401c1b4e5$b6aebd20$080a0a0a@g7n3g0>

Sorry Russ ... i believe I've also managed to send this to you personally
twice by accident ... technology defeats me ...

----- Original Message -----
From: "Russ Cox" <rsc@plan9.bell-labs.com>
To: <derek@chocolate-fish.com>; <getopt-sig@python.org>
Sent: Wednesday, February 13, 2002 9:21 PM
Subject: Re: [getopt-sig] there isn't really any discussion here

> That's not what I mean by ``optional arguments''.  What I mean is some
people
> want to define a flag -c that might or might not take an argument.
> In this case, it introduces an ambiguity.  If -c takes an optional
argument,
> does "foo -cd" mean "foo -c -d" or "foo -c d".  What if I want to specify
> -c without an argument but then have a normal command-line argument
> follow?  If you've got long options with their '--option=argument' syntax,
> there's no ambiguity to making the '=argument' not optional.

Aah I understand what you mean.  I agree, I really do not like this
possibility either, not just because of the parsing ambiguity.

> Then I don't see why it's not a command line argument.  Why not just
> define that the first thing on the command line after the options
> is the currency?  By your scheme, I would have to say
> "ssh --host=mycomputer" instead of "ssh mycomputer".
> I don't see the point of using option syntax when the option
> isn't really optional.

This is true, but only really when there is *one* such option.  What if it
needs a host name and a port (and it cant guess either, lets pretend ssh has
no standard port !) ?  Then using mandatory arguments they can go ssh -H
host -P port or ssh -P port -H host.  Without they *have* to go ssh host
port ... ie they *have* to know the predefined order before they execute the
script ... I like to enter arguments as they come to me (and i have  random
brain ;-)).

Ok this isnt a good example, every knows host comes before port but often
the arguments have little orderal relationship to each other.  Futher, it
lets you leave the remaining command line arguments for other more
traditional uses, such as a list of file names.

Des.






From derek@chocolate-fish.com  Wed Feb 13 23:33:39 2002
From: derek@chocolate-fish.com (Derek Harland)
Date: Wed, 13 Feb 2002 23:33:39 -0000
Subject: [getopt-sig] there isn't really any discussion here
References: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com> <004001c1b4d2$acec8c80$080a0a0a@g7n3g0> <20020213212346.GA2555@gerg.ca>
Message-ID: <00a101c1b4e6$de81a280$080a0a0a@g7n3g0>

----- Original Message -----
From: "Greg Ward" <gward@python.net>

[defns snipped --- I'll agree those]

>   required option
>     an option that must be supplied on the command-line; this is
>     an oxymoron and I personally consider it poor UI design.
>     (I gather Russ Cox agrees with me.)  Any getopt replacement should
>     make it fairly easy to implement required options, though, because
>     lots of people want them.

Its only an oxymoron as you describe it as "option".  I think of it this way
... "options" often come in a few flavours which are analgous to a GUI.  A
boolean option (eq python -i) is like a check box, its true or its false.  A
required option is often more like a radio button ... there can be lots of
groups of them and you must select *something* for each group.  Maybe I dont
wrap my brain correctly, but many scripts in my domain often require such
inputs, and I dont want to force the user to make them positional, and hence
dependent on order, i dont think you can trust users to do that.  Users will
always be invoking --help just to see the order. (frankly, its really that I
dont trust *myself* to get them in order)

The only "optional part" here is the value, the *parameter* itself is
required.  I think if you replace the word option with parameter its more
meaningful.

But I think this is just quibbling over crumbs.

Des.




From derek@chocolate-fish.com  Wed Feb 13 23:58:47 2002
From: derek@chocolate-fish.com (Derek Harland)
Date: Wed, 13 Feb 2002 23:58:47 -0000
Subject: [getopt-sig] my current argument parser
References: <0dfed672c11c9ac0990608e89b59f041@plan9.bell-labs.com>
Message-ID: <00c201c1b4ea$61338d80$080a0a0a@g7n3g0>

----- Original Message -----
From: "Russ Cox" <rsc@plan9.bell-labs.com>

Here are comments, you can rate their helpfulness !

> This illustrates all the important points:
>
> * arg is a generator that returns the next argument.
>   >>>The behavior of the generator depends on the behavior
>   of the body of your for loop.<<<  This is why you don't have
>   to tell the ArgParser about your options a priori.  If an option
>   has an argument, you call arg.nextarg() to fetch it.  (If you
>   want an option to have two arguments, call arg.nextarg twice.)

It might be nicer to be able instead to go a,b = arg.get(2),
if you know there are two values.  What about values = arg.consume() or
arg.getall() to consume all parameters to the next "argument" ?

ie given -I its iterative not optional -b wibble

you can go if o == 'I': words = args.consume() ??

> * Long options begin with -- and take arguments from an optional
>   ``=ARG'' suffix.  For example,
>   foo --long=bar

If we remove the optionality of long arguments then you can support
--long bar also.  Some people like this as it is at least consistent with
arguments to short options as well.  Personally I don't.

Des.




From rsc@plan9.bell-labs.com  Thu Feb 14 00:34:47 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Wed, 13 Feb 2002 19:34:47 -0500
Subject: [getopt-sig] my current argument parser
Message-ID: <7b542b0aab7ca99c618f159d5d89b88e@plan9.bell-labs.com>

Thanks for the comments.

> It might be nicer to be able instead to go a,b = arg.get(2),
> if you know there are two values.  What about values = arg.consume() or
> arg.getall() to consume all parameters to the next "argument" ?
> 
> ie given -I its iterative not optional -b wibble
> 
> you can go if o == 'I': words = args.consume() ??

args.consume() bothers me for the same reason that
optional arguments bother me.

Since having more than one argument to an option
is such a rare case, I'm not convinced that arg.get(2)
is really worthwhile.  If it was really common in a program,
I might subclass the parser.

> If we remove the optionality of long arguments then you can support
> --long bar also.  Some people like this as it is at least consistent with
> arguments to short options as well.  Personally I don't.

What do other tools do here?  I never use long options
so I don't really know what the standard behavior is.

In response to another message, I can understand the
desire to have mandatory options when configuration
is really a sequence of key-value pairs.  But I'm not convinced
the parser has any business checking such things.  I'd rather
it be a parser and add a check in the program.

Russ



From piers@cs.su.oz.au  Thu Feb 14 01:03:58 2002
From: piers@cs.su.oz.au (Piers Lauder)
Date: Thu, 14 Feb 2002 12:03:58 +1100
Subject: [getopt-sig] my vote is for automatic usage message generation
Message-ID: <1013648640.49.289740084@cs.usyd.edu.au>

While I agree with Russ Cox that the declaration of arguments should
(and can) be simple, and, as a programmer, I really appreciate the clean
interface he demonstrates, my overriding need is for automatic generation
of usage messages (the response to --help).

I would argue that parsing arguments has never been a serious problem,
merely a tedious problem.

All the serious problems that I've experienced with argument parsing
have come from incorrect (out-of-date) usage messages confusing users.

And program invoking users are those who matter here, the arguments are
their interface to the programs we write.

Piers Lauder.





From rsc@plan9.bell-labs.com  Thu Feb 14 03:11:38 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Wed, 13 Feb 2002 22:11:38 -0500
Subject: [getopt-sig] my vote is for automatic usage message generation
Message-ID: <b632854d200346e7f086fcd112722c87@plan9.bell-labs.com>

I'm torn about this.

I agree in principle -- it would be wonderful if we could arrange
things so that the code and the documentation automatically agree.

Even if I concede that full documentation belongs in the usage message
(which I'm not sure about), the price appears to be that you can't
customize the documentation as much as might be most helpful for the
users.  For example, below is output of GNU grep --help.  Perhaps the
getopt parser that GNU grep is using generated the message
automatically, but I doubt it.  I think that humans formatting the
usage messages still holds the best hope for very good usage messages.

As another example, consider sed's property that the presence of some
options (-e, -f) changes the interpretation of the remaining
arguments.  It's going to be hard to work that into an automatic usage
message-generation framework.

Finally, just because the parser knows about the option doesn't mean
that the program consults it.  I've seen far too many getopt format
strings that specify old arguments that have fallen out of use and
simply weren't removed from the string.  Automatic parser-generated
documentation doesn't help here.

I'm just not convinced that keeping the parser and the documentation
consistent completely solves the documentation problem; while it often
generates reasonable documentation, it makes it hard to write very
good documentation, and there's still no guarantee that the
documentation is correct (although I admit it's likely closer).  If
you write very good documentation by overriding the usage string,
you've lost the benefit and paid the price in (lack of) code clarity.

Russ

=====

Usage: grep [OPTION]... PATTERN [FILE] ...
Search for PATTERN in each FILE or standard input.
Example: grep -i 'hello world' menu.h main.c

Regexp selection and interpretation:
  -E, --extended-regexp     PATTERN is an extended regular expression
  -F, --fixed-strings       PATTERN is a set of newline-separated strings
  -G, --basic-regexp        PATTERN is a basic regular expression
  -e, --regexp=PATTERN      use PATTERN as a regular expression
  -f, --file=FILE           obtain PATTERN from FILE
  -i, --ignore-case         ignore case distinctions
  -w, --word-regexp         force PATTERN to match only whole words
  -x, --line-regexp         force PATTERN to match only whole lines
  -z, --null-data           a data line ends in 0 byte, not newline

Miscellaneous:
  -s, --no-messages         suppress error messages
  -v, --invert-match        select non-matching lines
  -V, --version             print version information and exit
      --help                display this help and exit
      --mmap                use memory-mapped input if possible

Output control:
  -b, --byte-offset         print the byte offset with output lines
  -n, --line-number         print line number with output lines
  -H, --with-filename       print the filename for each match
  -h, --no-filename         suppress the prefixing filename on output
  -q, --quiet, --silent     suppress all normal output
      --binary-files=TYPE   assume that binary files are TYPE
                            TYPE is 'binary', 'text', or 'without-match'.
  -a, --text                equivalent to --binary-files=text
  -I                        equivalent to --binary-files=without-match
  -d, --directories=ACTION  how to handle directories
                            ACTION is 'read', 'recurse', or 'skip'.
  -r, --recursive           equivalent to --directories=recurse.
  -L, --files-without-match only print FILE names containing no match
  -l, --files-with-matches  only print FILE names containing matches
  -c, --count               only print a count of matching lines per FILE
  -Z, --null                print 0 byte after FILE name

Context control:
  -B, --before-context=NUM  print NUM lines of leading context
  -A, --after-context=NUM   print NUM lines of trailing context
  -C, --context[=NUM]       print NUM (default 2) lines of output context
                            unless overridden by -A or -B
  -NUM                      same as --context=NUM
  -U, --binary              do not strip CR characters at EOL (MSDOS)
  -u, --unix-byte-offsets   report offsets as if CRs were not there (MSDOS)

=====

Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...

  -n, --quiet, --silent
                 suppress automatic printing of pattern space
  -e script, --expression=script
                 add the script to the commands to be executed
  -f script-file, --file=script-file
                 add the contents of script-file to the commands to be executed
      --help     display this help and exit
  -V, --version  output version information and exit

If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret.  All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.



From doug@hellfly.net  Thu Feb 14 04:31:59 2002
From: doug@hellfly.net (Doug Hellmann)
Date: Wed, 13 Feb 2002 23:31:59 -0500
Subject: [getopt-sig] my vote is for automatic usage message generation
In-Reply-To: <1013648640.49.289740084@cs.usyd.edu.au>
References: <1013648640.49.289740084@cs.usyd.edu.au>
Message-ID: <200202140431.XAA01916@branagan.hellfly.net>

On Wednesday 13 February 2002 20:03, Piers Lauder wrote:
> While I agree with Russ Cox that the declaration of arguments should
> (and can) be simple, and, as a programmer, I really appreciate the clean
> interface he demonstrates, my overriding need is for automatic generation
> of usage messages (the response to --help).

You might be interested in my CommandLineApp class.  If you create a new 
application by subclassing from it, you can then define a main method and 
option handlers for the options you want to provide.  The base class does all 
the boring work for you.

It automatically builds the arguments to getopt, calls the parser, and 
dispatches your handlers when options are encountered on the command line.  
And the other nice thing it does is provide -h and --help handling 
automatically based on docstrings provided with the handler methods.  There 
are a couple of class attributes that can be used to provide descriptions of 
the arguments (non-option parameters on the command line), as well as 
examples of how to use the program.

http://www.hellfly.net/PythonProjects/CommandLineApp

A quick example:

class MyApp(CommandLineApp):

    def optionHandler_a(self):
        "Short option, no argument."
        self.a = 1
    
    def optionHandler_b(self, withArgument):
        "Short option, with argument."
        self.b = withArgument

    def optionHandler_long_name(self):
        "Translates to --long-name"
        self.long = 1

    def optionHandler_long_with_arg(self, argumentValue):
        "Translates to --long-with-arg=argumentValue"
        self.long_arg = argumentValue

    def main(self, *args):
        for filename in args:
           print filename

This class would support -h, --help, -a, -b <val>, --long-name, and 
--long-with-arg=<val>.  Any extra values on the command line after the 
options would be passed as a vector to the main() method.

One of the reasons I joined the discussion list was to see what might be 
coming up as a getopt replacement, so I can incorporate support in this 
class.  I'm not sure if that's going to be necessary, but it could prove 
interesting.

Doug


From laurent.pointal@lure.u-psud.fr  Thu Feb 14 08:20:55 2002
From: laurent.pointal@lure.u-psud.fr (Laurent Pointal)
Date: Thu, 14 Feb 2002 09:20:55 +0100
Subject: [getopt-sig] A simple remark for the next Python command line option standard parser
Message-ID: <5.1.0.14.0.20020214091230.05f45ec0@webmail.lure.u-psud.fr>

A problem I encounter with standard getopt module is its absolute necessity 
to know all possible options (it raises an exception for unknown options).

I'm in the context of a tool module, with its **own** options. Application 
programmer use this module but dont have to care with its specific options 
(that may change as module evoluate).

What I need from the command line parser is the possibility to parse the 
command line to search for my tool options in my tool init function, and to 
ignore options which are targeted to other tools or to the main program 
(currently I rewrote my own -quick and not too dirty- sys.argv processing 
for my module need, but I would prefer use a standard module for that).
Sure, it may be nice "consume" options, or to mark them as processed, to 
identify in a time which are the options which are not used (and to signal 
it to users).

Good design.

A+

Laurent.


---
Laurent POINTAL - CNRS/LURE - Service Informatique Experiences
Tel/fax: 01 64 46 82 80 / 01 64 46 41 48
email  : laurent.pointal@lure.u-psud.fr  ou laurent.pointal@laposte.net



From tim.one@home.com  Thu Feb 14 08:32:07 2002
From: tim.one@home.com (Tim Peters)
Date: Thu, 14 Feb 2002 03:32:07 -0500
Subject: [getopt-sig] there isn't really any discussion here
In-Reply-To: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMECONMAA.tim.one@home.com>

[Russ Cox]
> ...
> It's not clear that one group is right and one group is wrong.

It doesn't even matter if one group *is* right and the other wrong:  the
only thing this SIG has to decide is which way is most Pythonic.
Unfortunately, that's harder than distinguishing good from evil <wink>.

> I think it's basically a difference of opinion about what constitutes
> the best approach to the problem.  Unfortunately I don't see any good
> way to sate both groups simultaneously.

We don't have to:  it's quite Pythonic to pick a winner, then move on.

Your approach is elegant to the point of transparency.  That's very
Pythonic.

OTOH, uniformity is supremely Pythonic, and Optik has several advantages
there:

+ Every Optik client generates the same kind of error msgs for a variety
  of errors (like missing argument, or unrecognized option name).  This
  is also true of getopt() clients, and is a real help to end users
  (they're not really fascinated by endless variations in common error
  messages).

+ Whenever Optik takes care of string-to-whatever conversion, "the
  rules" for exactly what can be converted and how it's done are the
  same across clients.  So saying, e.g., "integer argument" has a well-
  defined meaning across clients.  Also true of getopt clients, and
  again it's also valuable to have uniformity across programs in how
  failing conversions get reported.

+ Optik supplies policy for how options and argument values get
  reported back to you (as attributes of an object).  Current practice
  is a mess, storing option existence flags and argument values in as
  many diverse ways as getopt-loop authors have been able to dream up.
  This makes it harder to follow other peoples' code, and offering
  too much choice in what should be dull-as-dirt boilerplate code is
  downright anti-pythonic.  So are long function argument lists and/or
  masses of global variables.  People *should* often set up objects
  to carry options around, but laziness stops them.  Optik all but
  forces them into better practice, and that's quite appealing to
  the Pythonic mind.

+ Auto-generating a help msg in a uniform-across-clients way is also
  a value.  Ditto forcing all clients to have the same meaning for -h.

OTOH, everything else in Optik is useless fluff <wink>.



From s.keim  Thu Feb 14 11:18:30 2002
From: s.keim (s.keim)
Date: Thu, 14 Feb 2002 12:18:30 +0100
Subject: [getopt-sig] some comments about Optik
In-Reply-To: <20020213205204.GF2360@gerg.ca>
Message-ID: <93633BB2-213C-11D6-9026-0050E4A06334@laposte.net>

I'll try to give a follow up to your response and about a private mail 
Russ Cox sent to me:

>> Wouldn't it be easier for the user to directly use functions objects:
>> for sample:
> Or did you mean to put a type object there, like StringType or 'str'
> (same thing in Python 2.2)?  Two problems with that:
Yes that was what I was thinking about ( 'string' was a typo and I meant 
'str' actually)

>   * tricky to make it work the same way < 2.2 and >= 2.2:
>     if you want either 'str' or StringType to work in all versions
>     of Python (seems desirable to me)
This an important point for an independent package, not so true for a 
standard module which must enforce IMO the use of the language (and 
Python 2.2 is the language now).

>   * what about extensibility?  one of the major features of Optik's
>     type system is that its extensible: you could add a "file" or
>     "dir" or "outfile" type; some of those things correspond to
>     Python types or builtins, and some don't
Here we  only check string validity and generate value. I don't see  the 
extensibility problem: you can supply an user defined function instead 
of a type. For sample if you want to read a file key=value and convert 
it as a list of pairs:
def open_file(path):
	try:
		f = open(path,'r')
	except IOError:
		raise OptionError, "file "+path+" not found"
	return [ l.split'=' for l in f.readlines() ]   # well this would 
need more work in real life

> ...and that presupposes a name 'store_true' in each OptionParser
> instance, which means that adding an action requires subclassing both
> Option and OptionParser.
I haven't taken a look at the actual implementation, so ...

> Using strings to represent types and actions might be slightly more
> verbose, but I think it makes it easier to extend Optik: all you need to
> do is recognize one more string.
In fact, verbosity wasn't a matter for me, It was extension that I found 
quite complex.

I would give another example about the consequences of my proposals:
Suppose you provide to the 'action' method a subclass of the object 
returned by the 'type' method (with for instance an additional attribute 
_parse_info that contain ..surprisingly .. the extra informations from 
parser)
Then things start to become obvious:

For sample, you have defined an "append"  as appending data in a list. 
Now you have it for free:
L = []
Option("-f", action=L.append, type=str)

>> * required parameters
>> *groups
I agree that 'required option' is a non-sens. Later posts in the list 
have clarified the terminology so I won't say more about this, simply 
that my proposal wasn't very good.

My last feeling is that Optik is the union of 3 tools:
- a command line parser
- a grammar checker / doc generator
- a data processor (the 'action' stuff)
Maybe that the famous two groups  would be happier if we had this 3 
tools split and made to cooperate well.

And this could give us a lot of flexibility:
- one could replace the command line parser by a configuration file 
parser for instance
- or a unix cmd line grammar checker by a dos cmd line grammar checker
- or maybe it's just a dream ;-)


seb



From pac1@tiac.net  Thu Feb 14 11:33:40 2002
From: pac1@tiac.net (Patrick Callahan)
Date: Thu, 14 Feb 2002 06:33:40 -0500
Subject: [getopt-sig] Separation of mechanics from semantics
In-Reply-To: <20020213212346.GA2555@gerg.ca>
References: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com> <004001c1b4d2$acec8c80$080a0a0a@g7n3g0> <20020213212346.GA2555@gerg.ca>
Message-ID: <200202141133.g1EBXVPP003199@relayhost.us.inter.net>

Re: [getopt-sig] there isn't really any discussion here
Then lets start some.

What is it people really are trying to do when they ask for the kind of 
option parsing features that raise objections.  

On Wednesday 13 February 2002 04:23, Greg Ward wrote:
<snip>
>   option argument
>     an argument that follows an option and which is closely associated
>     with that option, and consumed from the argument list when the
>     option is.  Often, option arguments may also be included in the same
>     argument as the option, eg.
>       ["-f", "foo"]
>     may be equivalent to
>       ["-ffoo"]
>
>     Some options never take an argument.  Some options always take an
>     argument.  Lots of people want an "optional option arguments"
>     feature, meaning that some options will take an argument if they see
>     it, and won't if they don't.  This is somewhat controversial.
>

What kinds of semantics do people want to represent by having an option with 
an optional option argument.  What would the difference be between 
"-x "and "-x value"?  When people talk about allowing options to have 
optional arguments do they usually want to have the option without the 
argument represent some default value or do they have some other use in mind?


>   positional argument
>     something leftover in sys.argv after options have been parsed, ie.
>     options and option arguments removed from the argument list.
>
>     (Note that Optik does not change sys.argv, but returns a copy of
>     it with only positional args left.  Other getopt replacements
>     might work like this, or they might not.)
>
>   required option
>     an option that must be supplied on the command-line; this is
>     an oxymoron and I personally consider it poor UI design.
>     (I gather Russ Cox agrees with me.)  Any getopt replacement should
>     make it fairly easy to implement required options, though, because
>     lots of people want them.
>

Why want them?  What do people have in mind when they request this in an 
option package? What kind of thinking is really behind the requests for this? 
Why do they keep coming up?  

I'm not seeking an answer to whether its a good or bad thing at this point, 
just trying to find out more about how some of us think about the command 
line and how it is actually being used.

-Pat


From harri.pasanen@trema.com  Thu Feb 14 14:03:43 2002
From: harri.pasanen@trema.com (Harri Pasanen)
Date: Thu, 14 Feb 2002 15:03:43 +0100 (CET)
Subject: [getopt-sig] Idle thoughts
Message-ID: <200202141403.PAA29004@mark.labs.trema.com>

Just for the record, I do prefer Russ' Plan 9 style approach to Optik,
namely the straight forward C style approach to it.  

Optik seems to have too much of a learning curve, being somewhat
non-intuitive with all this action/type/dest stuff.  Almost like learning a
new language.

---

Vote cast, I do have a question:  Can either Optik or Russ' stuff be used
confortably in libraries?  One of my gripes with getopt is that if you look
for arguments in some generic module, getopt complains about the arguments
it does not understand.  I'd like to have the ability to stack arguments,
for instance turn on some debugging stuff in some imported library if
certain options are present in the command like, silently ignoring others.

Disclaimer:  I wrote this in a rush, without being really having used
anything than getopt and thus not delving into Optik's features  to the
level it would merit.


-Harri




From shane@zope.com  Thu Feb 14 15:19:05 2002
From: shane@zope.com (Shane Hathaway)
Date: Thu, 14 Feb 2002 10:19:05 -0500
Subject: [getopt-sig] ObjectCLI
Message-ID: <3C6BD569.9030706@zope.com>

Hi everyone,

Here is an approach I'd like you to consider for parsing command-line 
arguments.  I used this approach in the "pma" project (an NNTP server 
built on ZODB--see the zodbex project at SourceForge.)

ObjectCLI is a class that lets you invoke methods of a Python object 
directly from the command line.  The concept is similar to Zope's 
publisher, which lets you call methods directly from the Web.  Say you 
created a class like this:

class ServerControl:
   """A proxy server."""

   def start(port):
     """Starts the server on the specified port."""

   def stop():
     """Stops the server."""


With two or three lines of glue, you can then invoke the methods of a 
ServerControl instance directly from the command line:

python server_control.py start --port=3128
python server_control.py stop

You can also get the documentation of ServerControl, primarily derived 
from the docstrings:

python server_control.py --help

And you can get more in-depth documentation about each method:

python server_control.py start --help


The author of the ServerControl class never has to use getopt().  All 
you have to do is expose an object to the command line via ObjectCLI. 
(ObjectCLI uses getopt() to do its work.)


Major benefits:

- You almost never have to look up any library documentation to write 
new command line utilities.  You just write a regular Python class.

- The command line arguments and the implementation are less likely to 
fall out of sync.

- The documentation comes directly from the docstrings.


What do you think?

Shane



From s.keim  Thu Feb 14 15:27:47 2002
From: s.keim (s.keim)
Date: Thu, 14 Feb 2002 16:27:47 +0100
Subject: [getopt-sig] ObjectCLI
In-Reply-To: <3C6BD569.9030706@zope.com>
Message-ID: <663E3DD0-215F-11D6-9026-0050E4A06334@laposte.net>

Le jeudi 14 f=E9vrier 2002, =E0 04:19 PM, Shane Hathaway a =E9crit :

> Hi everyone,
>
> Here is an approach I'd like you to consider for parsing command-line=20=

> arguments.  I used this approach in the "pma" project (an NNTP server=20=

> built on ZODB--see the zodbex project at SourceForge.)
> ObjectCLI is a class that lets you invoke methods of a Python object=20=

> directly from the command line.

Isn't it something that is already supported by the standard library ?
http://python.org/doc/current/lib/module-cmd.html


seb



From shane@zope.com  Thu Feb 14 15:38:36 2002
From: shane@zope.com (Shane Hathaway)
Date: Thu, 14 Feb 2002 10:38:36 -0500
Subject: [getopt-sig] ObjectCLI
References: <663E3DD0-215F-11D6-9026-0050E4A06334@laposte.net>
Message-ID: <3C6BD9FC.9010401@zope.com>

s.keim wrote:
> 
> Le jeudi 14 février 2002, à 04:19 PM, Shane Hathaway a écrit :
> 
>> Hi everyone,
>>
>> Here is an approach I'd like you to consider for parsing command-line 
>> arguments.  I used this approach in the "pma" project (an NNTP server 
>> built on ZODB--see the zodbex project at SourceForge.)
>> ObjectCLI is a class that lets you invoke methods of a Python object 
>> directly from the command line.
> 
> 
> Isn't it something that is already supported by the standard library ?
> http://python.org/doc/current/lib/module-cmd.html

No.  cmd is for writing interactive command interpreters.  ObjectCLI is 
for writing non-interactive command-line parsers.

Shane



From shane@zope.com  Thu Feb 14 15:51:04 2002
From: shane@zope.com (Shane Hathaway)
Date: Thu, 14 Feb 2002 10:51:04 -0500
Subject: [getopt-sig] ObjectCLI
References: <663E3DD0-215F-11D6-9026-0050E4A06334@laposte.net> <3C6BD9FC.9010401@zope.com>
Message-ID: <3C6BDCE8.1060301@zope.com>

Shane Hathaway wrote:
> s.keim wrote:
>> Isn't it something that is already supported by the standard library ?
>> http://python.org/doc/current/lib/module-cmd.html
> 
> No.  cmd is for writing interactive command interpreters.  ObjectCLI is 
> for writing non-interactive command-line parsers.

Oops, just to clarify: ObjectCLI is for writing command line utilities 
and applications. (ObjectCLI *is* the parser, and you can certainly 
write an interactive utility.)

Shane



From sopwith@redhat.com  Thu Feb 14 16:14:33 2002
From: sopwith@redhat.com (Elliot Lee)
Date: Thu, 14 Feb 2002 11:14:33 -0500 (EST)
Subject: [getopt-sig] Re: ANNOUNCE: getopt-sig created
In-Reply-To: <mailman.1013638982.13521.clpa-moderators@python.org>
Message-ID: <Pine.LNX.4.44.0202141041330.23170-100000@devserv.devel.redhat.com>

On Tue, 12 Feb 2002, Greg Ward wrote:

> If you just want to state an opinion (eg. "Optik rules", "Optik sucks",
> "Russ' idea looks good to me"), please post it to getopt-sig@python.org
> -- you don't have to join the list to do that.

Optik sucks. Use python-popt.

(hey, you asked :)

popt is a C library that is used by rpm, gnome, etc. for the same purpose.  
There is a python-popt module used by some of the RH python utilities.

I'm pretty certain that nothing else out there can match the maturity and
feature set of popt. While I don't doubt Gregory's intentions, and I can
see pythonisms that make Optik uniquely suited to python, I don't see any
way for Optik to reflect the aggregate experience that has polished popt
into The Best For C Programs (and at least half-decent for Python ones :).

I am less certain that python would want to add a mandatory dependency on
a C library that is not installed by default on many of the systems python
supports.

If it were up to me, I would just keep getopt.py (which I personally
happen to use for my python programs that don't need complex cmdline
processing), and add poptmodule in to be built only if libpopt is
available.

Note of warning, whatever you do, stay away from glibc's argp routines....
-- Elliot
The early bird may get the worm, but the second mouse gets the cheese.



From gward@python.net  Thu Feb 14 18:22:07 2002
From: gward@python.net (Greg Ward)
Date: Thu, 14 Feb 2002 13:22:07 -0500
Subject: [getopt-sig] A simple remark for the next Python command line option standard parser
In-Reply-To: <5.1.0.14.0.20020214091230.05f45ec0@webmail.lure.u-psud.fr>
References: <5.1.0.14.0.20020214091230.05f45ec0@webmail.lure.u-psud.fr>
Message-ID: <20020214182207.GA2681@gerg.ca>

On 14 February 2002, Laurent Pointal said:
> A problem I encounter with standard getopt module is its absolute necessity 
> to know all possible options (it raises an exception for unknown options).
> 
> I'm in the context of a tool module, with its **own** options. Application 
> programmer use this module but dont have to care with its specific options 
> (that may change as module evoluate).

That was one of the motivating factors behind Optik.  The way it works
is this:

  * you define your own OptionParser subclass with its own
    STANDARD_OPTIONS class attribute; this is a list of Option
    instances

  * applications instantiate your OptionParser subclass, and then
    add_option() any of their own options

With an iterative model, things would be different: the "base" module
would probably provide a function that you call on each iteration of the
option-parsing loop, which just looks for the base module's own options.
Russ, does this make sense?

        Greg
-- 
Greg Ward - Python bigot                                gward@python.net
http://starship.python.net/~gward/
The world is coming to an end.  Please log off.


From rsc@plan9.bell-labs.com  Thu Feb 14 19:47:11 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Thu, 14 Feb 2002 14:47:11 -0500
Subject: [getopt-sig] A simple remark for the next Python command line option standard parser
Message-ID: <a979013f58edf13df3441878fda435f2@plan9.bell-labs.com>

> That was one of the motivating factors behind Optik.  The way it works
> is this:
> 
>   * you define your own OptionParser subclass with its own
>     STANDARD_OPTIONS class attribute; this is a list of Option
>     instances
> 
>   * applications instantiate your OptionParser subclass, and then
>     add_option() any of their own options
> 
> With an iterative model, things would be different: the "base" module
> would probably provide a function that you call on each iteration of the
> option-parsing loop, which just looks for the base module's own options.
> Russ, does this make sense?

I don't fully understand this approach.  What if I have two
modules I want to allow to take options from the command
line?  There's no multiple inheritance, so how do I let them
both do this?

I'd be much happier with modules providing an installoptions
function that adds to the option list somehow.

In the dict framework I threw out here before, you could do

	dict = {}
	module.installoptions(dict)
	module2.installoptions(dict)
	...
	add your own options to dict
	parseoptions(dict)

Or it could be implicit during module loading -- I can't decide
if that's a good thing.  In general, letting modules install
their own options is only going to be okay if it's used sparingly.

Russ



From pac1@tiac.net  Fri Feb 15 02:53:49 2002
From: pac1@tiac.net (Patrick Callahan)
Date: Thu, 14 Feb 2002 21:53:49 -0500
Subject: [getopt-sig] Separation of mechanics from semantics
In-Reply-To: <AB532854-215B-11D6-9026-0050E4A06334@laposte.net>
References: <AB532854-215B-11D6-9026-0050E4A06334@laposte.net>
Message-ID: <200202150253.g1F2rgPP020423@relayhost.us.inter.net>

If a set of command line options form a "language", Optik, as a lexical 
scanner limits the language to a small number of specific forms whose 
grammar can be expressed as a single simple unordered list.

Optik simply looks at the tokens in the stream of command line artifacts and 
acts on each one in turn.  It contains no model of sequencing or combination 
for the options and simply checks to ensure that each token or sequential 
pair of tokens is a word in the "language", a word being either a single 
option or an option value pair.

More complex command line option models, such as the one used in tar 
introduce  additional layers to the specification of the command line 
"language", one that cannot be expressed or processed so simply.  They check 
more than if the token is in the language or not.  It is at this level that 
checks for combinations of tokens or such requirements as "one of c, r, t, u 
or x must be supplied" is done.  I don't think it could be cleanly integrated 
with the lower level scanning done by optik.

Any mechanism which must use a more complex model must have some sort of 
underlying parsing mechanism to deal with the tokens one at a time.  That 
said,  are there ways we could layer the more complex parsing model on optik?
What would the lexical rules of such a mechanism look like?  

There's a number of challenges in devising such a mechanism:

o  What kind of expression could be used to convey the rules? 
o Could the basic parsing rules for optik be easily derived  or would they 
need to be stated separately?
o  Would the mechanism be general enough to be widely useful?
o  Could the mechanism be misused,? 

Here's that pesky Tar example again:

> But again, this is only an example, not  an argument in favor or against
> supporting required parameters or optional arguments.
>
>
>
> man tar
>
> TAR(1)                      System Reference Manual
> TAR(1)
>
> NAME
>       tar - tape archiver
>
> SYNOPSIS
>       tar [-]{crtux}[befhmopvwzHLPXZ014578] [archive] [blocksize] [-C
> directory
>       ] [-s replstr ] file1 [file2...]
>
> <snip...>
>
>       One of the following flags must be present:
>
>       -c            Create new archive, or overwrite an existing archive,
>                     adding the specified files to it.
>
>     <snip....>
>    In addition to the flags mentioned above, any of the following flags
> may
>       be used:
>
>       -b blocking factor
>                     Set blocking factor to use for the archive, tar uses
> 512
>     <snip ...>


From s.keim  Fri Feb 15 08:42:30 2002
From: s.keim (s.keim)
Date: Fri, 15 Feb 2002 09:42:30 +0100
Subject: [getopt-sig] A simple remark for the next Python command line option standard parser
In-Reply-To: <a979013f58edf13df3441878fda435f2@plan9.bell-labs.com>
Message-ID: <F2B87F58-21EF-11D6-9026-0050E4A06334@laposte.net>

Le jeudi 14 f=E9vrier 2002, =E0 08:47 PM, Russ Cox a =E9crit :

> Or it could be implicit during module loading -- I can't decide
> if that's a good thing.  In general, letting modules install
> their own options is only going to be okay if it's used sparingly.

This could have the advantage to modify the option management into a=20
module without breaking scripts that use this module.

But I find it quite dangerous, name clashes will be  quite hard to=20
manage.
Or maybe you was thinking about a group mechanism like in XWindow=20
applications command lines, but isn't this overkill ?

seb



From dittmar@snafu.de  Fri Feb 15 11:20:03 2002
From: dittmar@snafu.de (dittmar@snafu.de)
Date: Fri, 15 Feb 2002 11:20:03 GMT
Subject: [getopt-sig] Random thoughts
Message-ID: <E16bgPL-00046g-00@smart.eusc.inter.net>

Thought 1

+1 for Optik vs. Iterator
- less typing for the simple cases
- it's easier to combine options (application: multiple
    scripts connecting to a database can share option 
    definitions related to connections and add their own
    options)
- all the information for one option is in one place
    * name (s)
    * initialization/default
    * argument processing
    * check at end of processing (not yet in Optik)

There is the occasional use for an iterator 
(e.g. -o <outfile> <infile> -o <outfile> <infile> ...)
but as I don't care much for combined options (-cfd instead of -c -f -d),
I simply don't support them and then writing an iterator is trivial.

Thought 2

Would the following be a violation of 
"explicit is better than implicit" or an application of
"don't repeat yourself"?
Optik:
the default for "dest" is the value of the 
long option without '--'
the default for "type" is deduced from the type of the "default" value

Thought 3

Should there be support for more sources of arguments?
* @filename in argument list => will expand contents of
    filename into argument list
* Optik (env = varname) will prepend additional options from
    os.environ.get (varname, ''), it's the name of a 
    environment variable and not a string because
    that name should be generated into the documentation

Thought 4

I'm not sure that adding new actions through subclassing
is the way to go as this makes it a bit cumbersome to add
actions from different sources.

Thought 5

My own option lib can be called as main and produces then a 
skeleton script:
#!/usr/bin/env pythhon
# dummy.py

def main (options, args):
    for arg in args:
        pass

def _options ():
    return [
        # (optstring, varname, typechar, default, help)
        ]

if __name__ == '__main__':
    import optlib
    optlib.optMain2 (main, _options ())

The important part is the comment in _options, as it gives
a template how to fill in the various fields.

This could be extended by using additional arguments to transform
'--output=xyz -v --count=0'
to
parser.add_option ("??", "--output", action="store", dest="output",
    default="xyz")
parser.add_option ("-v", action="store_true", dest="v")
parser.add_option ("??", "--count", action="store", type="int",
    dest="count", default=0)

Thought 6

One application of 'optional argument to option':
script --log=x.log   writes to x.log
script --log         writes to default-file-name
script               writes no log

Not that I use this often enough to require it in a standard lib.

Daniel




From gward@python.net  Fri Feb 15 18:43:32 2002
From: gward@python.net (Greg Ward)
Date: Fri, 15 Feb 2002 13:43:32 -0500
Subject: [getopt-sig] Separation of mechanics from semantics
In-Reply-To: <200202141133.g1EBXVPP003199@relayhost.us.inter.net>
References: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com> <004001c1b4d2$acec8c80$080a0a0a@g7n3g0> <20020213212346.GA2555@gerg.ca> <200202141133.g1EBXVPP003199@relayhost.us.inter.net>
Message-ID: <20020215184332.GA5610@gerg.ca>

On 14 February 2002, Patrick Callahan said:
> What kinds of semantics do people want to represent by having an option with 
> an optional option argument.  What would the difference be between 
> "-x "and "-x value"?  When people talk about allowing options to have 
> optional arguments do they usually want to have the option without the 
> argument represent some default value or do they have some other use in mind?

I'm one of the people who has, on occasion, wanted to have optional
option arguments.  The only example I can think of offhand is in the
Distutils, specifically the --home option to the "install" command.

Background: usually when you install something with the Distutils, it
goes to the standard system-wide location for third-party Python
modules, eg. /usr/local/lib/python2.2/site-packages on Unix.  (Same idea
on Windows and Mac OS, but the layout is of course different.)
Sometimes that's not what you want, and my guess was that the most
common "non-standard" installation would be to your home directory (at
least on Unix).  The way I *wanted* this to work was

  setup.py install             # install to system dir
  setup.py install --home      # install to my home dir (~/lib/python)
  setup.py install --home=/tmp # install as though my home dir were
                               # /tmp (/tmp/lib/python)

But since the Distutils use getopt underneath, an option must either
take an argument or not.  So the second version has to be spelled like
this:

  setup.py install --home=~

...and I had to write code that specifically handles "~", because not
all shells will do it in this context (ie. not at the beginning of a
word).  (Of course, you could also write "--home=$HOME" or "--home ~",
but I wanted "--home=~" to just work.)

This is not a big deal, and I don't think it kills the usability of the
Distutils.  It's just a very small wart that I once thought could be
cured by having optional option arguments.  The more I think about it,
the less convinced that they're a good idea.  At some point, I'll
probably try to implement them experimentally in Optik, and that will
decide me.  Most likely the extra complexity and syntactic ambiguity
will kill the idea.

        Greg
-- 
Greg Ward - Unix bigot                                  gward@python.net
http://starship.python.net/~gward/
Drive defensively -- buy a tank.


From gward@python.net  Fri Feb 15 18:48:55 2002
From: gward@python.net (Greg Ward)
Date: Fri, 15 Feb 2002 13:48:55 -0500
Subject: [getopt-sig] Separation of mechanics from semantics
In-Reply-To: <200202150253.g1F2rgPP020423@relayhost.us.inter.net>
References: <AB532854-215B-11D6-9026-0050E4A06334@laposte.net> <200202150253.g1F2rgPP020423@relayhost.us.inter.net>
Message-ID: <20020215184855.GB5610@gerg.ca>

On 14 February 2002, Patrick Callahan said:
> More complex command line option models, such as the one used in tar
> introduce additional layers to the specification of the command line
> "language", one that cannot be expressed or processed so simply.  They
> check more than if the token is in the language or not.  It is at this
> level that checks for combinations of tokens or such requirements as
> "one of c, r, t, u or x must be supplied" is done.  I don't think it
> could be cleanly integrated with the lower level scanning done by
> optik.

I don't think "tar" -- or "find" or "dd", for that matter -- are
examples of user interface design that we should be encouraging (or even
enabling) people to perpetrate.  Some UIs are just plain bad, and if
Optik discourages people from making bad UIs, that's all the better.

However, I suspect you could pull of something like tar's clunky and
painful interface using callbacks.  I'm sure Russ Cox will jump in and
say that an iterative model is cleaner for this sort of thing, and he's
absolutely right: callbacks are themselves clunky and painful.  They're
an escape hatch for occasions where you just don't fit in Optik's box,
not something you should use in every program.  (Optik's box happens to
feel fairly spacious and accomodating to me, so I use callbacks quite
rarely.)

Bottom line, if you want to create a botched UI like "tar", you're free
to do so -- but you'll have to implement your own argument parser.  I
suspect Russ' model would make more sense in that context.

        Greg
-- 
Greg Ward - just another /P(erl|ython)/ hacker          gward@python.net
http://starship.python.net/~gward/
Time flies like an arrow; fruit flies like a banana.


From gward@python.net  Fri Feb 15 18:52:09 2002
From: gward@python.net (Greg Ward)
Date: Fri, 15 Feb 2002 13:52:09 -0500
Subject: [getopt-sig] there isn't really any discussion here
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMECONMAA.tim.one@home.com>
References: <293adac0c149824f29cfd878c56b7b14@plan9.bell-labs.com> <LNBBLJKPBEHFEDALKOLCMECONMAA.tim.one@home.com>
Message-ID: <20020215185209.GC5610@gerg.ca>

On 14 February 2002, Tim Peters said:
> + Auto-generating a help msg in a uniform-across-clients way is also
>   a value.  Ditto forcing all clients to have the same meaning for -h.

It would be neat if there was a way for clients to provide helpful hints
for occasions where Optik botches the formatting of the help, as I'm
sure could happen.

Also, Optik doesn't *force* all clients to have -h/--help; you can
always create your own OptionParser subclass with no standard help
option.  But it's awkward and difficult to do so, which is just as good
as force.

        Greg
-- 
Greg Ward - Unix bigot                                  gward@python.net
http://starship.python.net/~gward/
Reality is for people who can't handle science fiction.


From gward@python.net  Fri Feb 15 19:09:53 2002
From: gward@python.net (Greg Ward)
Date: Fri, 15 Feb 2002 14:09:53 -0500
Subject: [getopt-sig] Random thoughts
In-Reply-To: <E16bgPL-00046g-00@smart.eusc.inter.net>
References: <E16bgPL-00046g-00@smart.eusc.inter.net>
Message-ID: <20020215190953.GD5610@gerg.ca>

On 15 February 2002, dittmar@snafu.de said:
> Thought 2
> 
> Would the following be a violation of 
> "explicit is better than implicit" or an application of
> "don't repeat yourself"?
> Optik:
> the default for "dest" is the value of the 
> long option without '--'

Someone else already suggested that, and I'm open to it.  I just peeked
at the code and it should be easy to slip in.

> the default for "type" is deduced from the type of the "default" value

No, that's too sneaky and implicit.  I don't like it.

However, I am thinking of making the default type "string" -- ie. if
Optik expected a type (action="store" or action="append" or dest
supplied), and you didn't give one, it assumes type="string".  That's
also implicit behaviour, so potentially unPythonic.  But it's awfully
convenient, since most options are string options.  Opinions?

> Thought 3
> 
> Should there be support for more sources of arguments?
> * @filename in argument list => will expand contents of
>     filename into argument list

No.  If you want this, implement an OptionParser subclass.  If that's
painful, let me know how OptionParser needs to be refactored.

> * Optik (env = varname) will prepend additional options from
>     os.environ.get (varname, ''), it's the name of a 
>     environment variable and not a string because
>     that name should be generated into the documentation

Maaaaybe, but I'm cool to it.  Same advice as above.

Basically, I want to stop adding features to Optik.  If you want a
feature, implement it by subclassing.  If subclassing is painful, then
refactoring is called for and I want to hear about it.

> Thought 6
> 
> One application of 'optional argument to option':
> script --log=x.log   writes to x.log
> script --log         writes to default-file-name
> script               writes no log

Yeah, good example -- easier to grasp then my Distutils example.

        Greg
-- 
Greg Ward - programmer-at-large                         gward@python.net
http://starship.python.net/~gward/
This quote intentionally left blank.


From tim.one@home.com  Fri Feb 15 20:07:10 2002
From: tim.one@home.com (Tim Peters)
Date: Fri, 15 Feb 2002 15:07:10 -0500
Subject: [getopt-sig] there isn't really any discussion here
In-Reply-To: <20020215185209.GC5610@gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEKKNMAA.tim.one@home.com>

[Tim]
> + Auto-generating a help msg in a uniform-across-clients way is also
>   a value.  Ditto forcing all clients to have the same meaning for -h.

[Greg Ward]
> It would be neat if there was a way for clients to provide helpful hints
> for occasions where Optik botches the formatting of the help, as I'm
> sure could happen.

It's quite possibly better if clients learn to write option help strings such that Optik
doesn't botch the formatting.  Uniformity across apps is a real value to end users (if
the "tar" example showed anything, it's in part why it takes years to become truly fluent
with the Unix toolset <0.6 wink>).

> Also, Optik doesn't *force* all clients to have -h/--help; you can
> always create your own OptionParser subclass with no standard help
> option.  But it's awkward and difficult to do so, which is just as good
> as force.

Well, you can subclass Random and Queue too, but not one in a thousand is obsessed enough
to bother.  For mass use, a framework is valuable to the extent that most programmers can
and do use it without extending.  Most people want something that cracks their
(well-designed <wink>) cmdline out of the box.  Optik looks very good on that count, and
anyone overriding default -h behavior is at best anti-social.



From jonathan@onegoodidea.com  Fri Feb 15 22:23:50 2002
From: jonathan@onegoodidea.com (Jonathan Hogg)
Date: Fri, 15 Feb 2002 22:23:50 +0000
Subject: [getopt-sig] DPyGetOpt
Message-ID: <B8933AF6.4E68%jonathan@onegoodidea.com>

Hi,

Just thought I'd drop a note to mention Bill Bumgarner's DPyGetOpt. As far
as I'm aware, it has been out of active development for a long while. But
the old code is still usable and the Arusha project has modified it to use
re instead of regex and uses it for options parsing.

    Bill Bumgarner <mailto:bbum@friday.com>
    Arusha <http://ark.sourceforge.net>

Jonathan

-- 
jonathan hogg, one good idea ltd, 131 queen margaret dr., glasgow g20 8pd
http://www.onegoodidea.com/ tel:+44-(0)7976-614338 fax:+44-(0)7970-537451



From pac1@tiac.net  Fri Feb 15 23:02:01 2002
From: pac1@tiac.net (Patrick Callahan)
Date: Fri, 15 Feb 2002 18:02:01 -0500
Subject: [getopt-sig] Separation of mechanics from semantics
In-Reply-To: <20020215184855.GB5610@gerg.ca>
References: <AB532854-215B-11D6-9026-0050E4A06334@laposte.net> <200202150253.g1F2rgPP020423@relayhost.us.inter.net> <20020215184855.GB5610@gerg.ca>
Message-ID: <200202152301.g1FN1nPP029171@relayhost.us.inter.net>

Greg wrote:
> Bottom line, if you want to create a botched UI like "tar", you're free
> to do so -- but you'll have to implement your own argument parser.  I
> suspect Russ' model would make more sense in that context.
>
>         Greg

Hmm.  Tar is botched....  Could we learn anything from an attempt to 
"redesign" tar's user interface?  Things are the way they are sometimes for 
reasons we can't see. Or sometimes its just botched.  Either way, I'm 
interested enough to discuss the fine points if you're up to it?

When you say botched, are you referring to the "required" option 
The first argument to tar must
       be one of the options: Acdrtux, followed by  any  optional
       functions.
or some other aspect of it?

The tar utility has a lot of capabilities built into a single utility is that 
the source of the complexity?  Should tar have been split into multiple 
commands?

I've no particular axe to grind on tar itself, but I'm really interested in 
your thought process on command line interfaces.

-Pat


From gward@python.net  Mon Feb 18 00:01:00 2002
From: gward@python.net (Greg Ward)
Date: Sun, 17 Feb 2002 19:01:00 -0500
Subject: [getopt-sig] Recent changes in Optik
Message-ID: <20020218000100.GA1006@gerg.ca>

Hi all --

based on some of the feedback and requests seen in the past week, I've
made some minor changes to Optik that, I think, will make it even easier
to use.

The most visible is that you can now get away without supplying a type
or a destination for many options.  If you don't supply a type and one
is needed, Optik uses "string"; if you don't supply a destination and
one is needed, Optik extracts a destination from your first long option,
or your first short option if none is supplied.

For example, the following is now valid:

  parser = OptionParser()
  parser.add_option("-f", "--file")

This defines an option whose action is to store a value of type string
in a variable 'file'.  (Well, really, in an instance attribute 'file' of
the Values instance that is created and returned by
parser.parse_args()).  Thus, for the most common kind of option -- the
kind that takes a single string and stores it somewhere, where later
occurences of the option override earlier occurences -- you don't *need*
to specify any keyword args at all.  (You should still supply a help
string, but it's not required.)

If you're really lazy -- or writing a really a quick hack -- you can
get away with

  parser.add_option("-a")

which stores a single string value to the variable 'a'.

The downside of this is that it enables programmers to be implicit
rather than explicit, and we all know that explicit is better than
implicit.  Personally, I will probably take advantage of the default
type a lot, but not the default destination.  YMMV.

Also, I've done some refactoring of the Option and OptionParser classes
that will make extending Optik (in certain directions) easier.  More on
that later.

This is all available by CVS only right now.  There might be an Optik
1.2.1 release soon.  If you're actively using Optik right now, you
should probably check out the CVS code and make sure I haven't screwed
anything up!  (Yes, I have tested all these changes, so you should be
fine.  If you have been subclassing Option and/or OptionParser in weird
ways, though, you might feel some pain.)  Go to optik.sourceforge.net
and follow the links...

        Greg
-- 
Greg Ward - Unix weenie                                 gward@python.net
http://starship.python.net/~gward/
I just read that 50% of the population has below median IQ!


From s.keim  Mon Feb 18 09:25:12 2002
From: s.keim (s.keim)
Date: Mon, 18 Feb 2002 10:25:12 +0100
Subject: [getopt-sig] Yet another parser proposal
In-Reply-To: <3C6BD9FC.9010401@zope.com>
Message-ID: <68C679F3-2451-11D6-849E-0050E4A06334@laposte.net>

#  quickcmd.py
#
# a quick and dirty implementation around Shane Hathaway idea:
# let's use python syntax for cmd line grammar specification.
#
# the cmd line must match a function prototype:
#      *options are converted into args with default value
#      *positional parameters are converted into args without default 
value
#      *use default value type of keyword arguments for type checking
#      *you can also have options without default value, if you provide
#       a callable object as keyword argument default value
#      *a None default value specify a flag (an option without value)
#      *the *kw protocol specify unlimited number of positional 
parameters
#      *you can handle multi value for option by using tupple as default 
value
#      *if you don't care about the number of values you can use list
#       note that the user will have to provide at least one value 
(maybe something
#       to improve) and that if the number of values provided is greater 
than the list
#       length, additional values will be left as strings.
#
# Well it wasn't easy to switch between function arguments and command 
line arguments
# in this explanation , but it's quite easy to use! isn't it?
#
from __future__ import generators
import sys, inspect, types
		
class unix_parser:
     """parsing is split from grammar checking, this allow to use
     alternative parsers for other syntax (eg DOS syntax)"""
     def __init__(self, usage, argv):
         self.argv0 = argv[0]
         self.argv = argv[1:]
         self.usage = usage
         self.synopsis = None

     def __iter__(self):
         while self.argv:         #positionnals
             token = self.argv.pop(0)
             if token[0]=='-':
                 break
             yield None, token
             token = None
         while token:        #options
             if token[:2] == '--':
                 s = token[2:].split('=')
             else:
                 s = [token[1]]
                 if token[2:]: s+= [token[2:]]
             key = s[0]
             value = s[1:]
             while self.argv:
                 token = self.argv.pop(0)
                 if token[0]=='-':
                     yield (key,value)
                     break
                 if token[0] == '=':
                     token=token[1:]
                 if token != '' :
                     value.append(token)
             else:
                 token = None
             yield (key, value)

     def error(self, msg=None):
         if msg:
             sys.stderr.write('argument error: '+msg+'\n')
         sys.stderr.write('\n'+"SYNOPSIS : "+self.argv0+' 
'+str(self.synopsis)+'\n')
         sys.stderr.write(self.usage)
         sys.stderr.flush()
         sys.exit(1)

     def set_synopsis(self, arg_name, varargs,  options, flags):
         def sign(k):
             return '-'*(1+(len(k)>1)) + k
         def _option(arg):
             key,value = arg
             s = '['+sign(key)
             if type(value) in (type(()), type([])):
                 s+= ' '
             else:
                 s+=  len(key)>1 and '=' or ' '
                 value = (value,)
             s+= ' '.join([callable(i) and '<'+i.__name__+'>' or str(i)  
for i in value])
             if type(value) == type([]): s+='...'
             return s+']'
         def _flag(key):
             return '['+sign(key)+']'
         h = ' '.join(arg_name)+' '
         if varargs:
             h+= '['+varargs+'...]'
         if 'help' not in options:
             h+='[--help]'
         h+= ''.join(map(_option,options.items()) + map(_flag,flags))
         self.synopsis = h + '\n'


class OptionError(Exception):
      def check(cls, condition, *args):
          if not condition:
              raise cls(*args)
      check = classmethod(check)

# for the type checking
class OptionTypeError(OptionError):
     def __init__(self, name, value, _type):
         self.name = name
         self.value = value
         self.type = _type
     def __str__(self):
         return "bad value for "+self.name+", "+self.value+ \
                " should be ("+self.type.__name__+")"

def _isdefault(v):
     if callable(v):
         return None
     if type(v) in (type(()),type([])):
         for i in v:
              if callable(i):
                 return None
     return 1

def _checktype(name, values, def_vals):
     unpack = None
     remain = []
     if type(def_vals) ==  type(()):
         OptionError.check(len(values)==len(def_vals), "bad number of 
values for "+name)
     elif type(def_vals) ==  type([]):
         l = min(len(values),len(def_vals))
         values,remain  = values[:l], values[l:]
         def_vals = def_vals[:l]
     else:
         def_vals = (def_vals,)
         unpack = 1
     data = []
     for v,d in zip(values, def_vals):
         if (callable(d)):
             creator = d
         else:
             creator = type(d)
         if creator == types.InstanceType:
             creator = def_val.__class__
         try:
             data.append(creator(v))
         except ValueError:
             raise OptionTypeError(name, v, creator)
     if unpack:
         return data[0]
     else:
         return data+remain

def _grammar(func):
      """return a grammar from function signature:
         return a tupple:
         ([pos args], varargs, {options=value}, {flags=None})
      """
      args, varargs, varkw, defaults = inspect.getargspec(func)
      if varkw : raise TypeError, "grammar function can't use **"+varkw
      l = len(args)-len(defaults)
      args, opts = args[:l], zip(args[l:],defaults)
      flags = {}; options = {}
      for key,value in opts:
          if value:
              options[key] = value
          else:
              flags[key] = None
      return args, varargs, options, flags

class quickcmd:
     def __init__(self):
         self.keywords=[]
         self.options={}

     def feed(self, func, line=sys.argv, Parser = unix_parser):
          """ cmd line analysys with func gramar """
          arg_names, varargs, def_options, def_flags = _grammar(func)
          #would it be beter to use module docstring instead?
          self.parser = parser = Parser(str(inspect.getdoc(func)), line)
          parser.set_synopsis(arg_names,varargs,def_options,def_flags)
          self.arg_names = arg_names, varargs

          options={}; positionnals=[]
          for key, value in parser:
              if key is None:
                  positionnals.append(value)
              else:
                  if value == []:
                      if key in def_flags:
                          options[key] = 1
                      elif key=='help':
                          parser.error()
                      elif key in def_options:
                          parser.error ("value required for option : 
"+key)
                      else:
                          parser.error ("unknown option : "+key)
                  else:
                      if key in def_options:
                          try:
                              options[key] = _checktype(key, value, 
def_options[key])
                          except OptionError, e:
                              parser.error(str(e))
                      elif key in def_flags:
                          parser.error ("option "+key+" can't have value")
                      else:
                          parser.error ("unknown option : "+key)


          #allow change to self.options by type checking functions
          #with this you could for sample let the user specify a file 
containing defaults
          def_options.update(self.options)
          def_options.update(options)
          self.options = def_options
          self.keywords.extend(positionnals)
          if len(self.keywords)!=len(arg_names) and not varargs:
              parser.error("bad number of positionnals arguments "
                           "(should be exactely "+str(len(arg_names))+")")
          elif len(self.keywords)<len(arg_names):
              parser.error("not enough positionnals arguments "
                           "(should be at least "+str(len(arg_names))+")")
          try:
              # going around a strange feature of python:
              # the following seems to not work when you have unlimited 
number of positionnals:
              # func(*positionnals, **options)
              l = len(arg_names)
              def_flags.update(def_options)
              names = inspect.getargspec(func)[0][l:]
              p = positionnals[:l] + [def_flags[n] for n in names] + 
positionnals[l:]
              func(*p)

          except OptionError,e:
              parser.error(str(e))
          for key,value in self.options.items():
              if not _isdefault(value):
                  del self.options[key]


     def checkargs(self,*_types):
         """this function must be used to check the positionnals 
arguments"""
         keywords = self.keywords
         size = len(_types)
         arg_names = self.arg_names[0]
         arg_names += [self.arg_names[1]] * (size-len(arg_names))
         result = []
         for name, value, _type in zip(arg_names, keywords, _types):
             try:
                 result.append(_checktype(name, [value], _type))
             except OptionTypeError, e:
                 self.parser.error(str(e))
         self.keywords = result + keywords[size:]

         def error(self, msg):
                 self.parser.error(msg)



#---------------------------------------------------------
# the following should go on your own module


# this is the function used to define cmd line grammar
def tester (spam, egs, xy=(1,1), ran='yes', splof=None, dest=[str], 
v=int, *files):
      """a description of your script

      spam    a positional argument (string)
      egs     another positional argument (float)
      files   specify unlimited positional arguments
      xy      an option with two integer values
      ran     an option with a string value
      splof   an option without value (a flag)
      v       a short option of type int but without default value
      dest    an option with unlimited number of strings values
      """

      #this is a good place for semantical checking:
      OptionError.check(ran in ['yes','no'],
                        " ran must be set to 'yes' or 'no'")
      OptionError.check((not splof) or v!=int ,
                        "v must be set if splof activated") #meaningful 
message ;)

# yes it's all
# hmm...wait, you still have to call the parser ;-)

cmd = quickcmd()
cmd.feed(tester, ['tester', 'f1', '89.5', '--xy', '45', '17'])
                #  , '--ran=no','--dest', '/tmp/chx', 
'/usr/slog','--splof','-v10'])

#I was unable to automate type checking-conversion for positional args
cmd.checkargs(str,float)
#maybe would it be better to have this values returned by the grammar 
function?

# it's really all :) , now you can use cmd.options and cmd.keywords
print '-'*50
print cmd.keywords
print cmd.options



From david@sleepydog.net  Mon Feb 18 15:08:49 2002
From: david@sleepydog.net (David Boddie)
Date: Mon, 18 Feb 2002 15:08:49 +0000
Subject: [getopt-sig] Parse arguments according to a usage message?
Message-ID: <20020218151030.1E8CD2AA69@wireless-084-136.tele2.co.uk>

Having read the messages about separating mechanics from semantics

http://mail.python.org/pipermail/getopt-sig/2002-February/000039.html

and generating usage messages automatically

http://mail.python.org/pipermail/getopt-sig/2002-February/000024.html

I wonder whether there is any mileage in using part of the class that I
mentioned in the "RFC: Option Parsing Libraries" thread in python-dev?

http://www-solar.mcs.st-and.ac.uk/~davidb/Software/Python/cmdsyntax/

Here's some abridged output from the test.py script included in the
cmdsyntax.zip archive found from the page above. Apologies for the amount
of output, but I'm trying to illustrate the capabilities of the parser.

The first example allows either the short or long form of switches/options
to be specified, although the parser doesn't know that -d and --directory
might be equivalent in this case.

  [davidb@bunsen CMDSyntax]$ test.py -d output myfile
  Syntax: ./test.py [(-d dir) | --directory=dir] infile
  Ordered optional arguments? [y/n] n

[Removed some output here...]

  First match found:

  dir     :       output
  infile  :       myfile
  -d      :       1

  [davidb@bunsen CMDSyntax]$ test.py --directory=output myfile
  Syntax: ./test.py [(-d dir) | --directory=dir] infile
  Ordered optional arguments? [y/n] n

[Removed some output here...]

  First match found:

  infile          :               myfile
  --directory     :               output


Arguments can be optional and may contain optional arguments of their
own.

  [davidb@bunsen CMDSyntax]$ test.py -a -b bernard
  Syntax: ./test.py [-a] [[-b badger] -c]
  Ordered optional arguments? [y/n] n

[Removed some output here...]

  Quick match with command line? [y/n] y
  No matches found

  [davidb@bunsen CMDSyntax]$ test.py -b bernard -c
  Syntax: ./test.py [-a] [[-b badger] -c]
  Ordered optional arguments? [y/n] n

[Removed some output here...]

  Quick match with command line? [y/n] y
  First match found:

  -b      :       1
  -c      :       1
  badger  :       bernard


Optional arguments placed together can be ordered arbitrarily without
disturbing mandatory arguments which must occur in the correct order.

  [davidb@bunsen CMDSyntax]$ test.py input -b hello -a world output
  Syntax: ./test.py infile [-a abacus] [-b binary] outfile [-c computer]
  Ordered optional arguments? [y/n] n

[Removed some output here...]

  Quick match with command line? [y/n] n
  First match found:

  infile  :       input
  binary  :       hello
  abacus  :       world
  -b      :       1
  -a      :       1
  outfile :       output

Is this sort of thing useful, or relevant to the discussion?

David

________________________________________________________________________
This email has been scanned for all viruses by the MessageLabs SkyScan
service. For more information on a proactive anti-virus service working
around the clock, around the globe, visit http://www.messagelabs.com
________________________________________________________________________


From gward@python.net  Mon Feb 18 16:07:16 2002
From: gward@python.net (Greg Ward)
Date: Mon, 18 Feb 2002 11:07:16 -0500
Subject: [getopt-sig] Separation of mechanics from semantics
In-Reply-To: <200202152301.g1FN1nPP029171@relayhost.us.inter.net>
References: <AB532854-215B-11D6-9026-0050E4A06334@laposte.net> <200202150253.g1F2rgPP020423@relayhost.us.inter.net> <20020215184855.GB5610@gerg.ca> <200202152301.g1FN1nPP029171@relayhost.us.inter.net>
Message-ID: <20020218160716.GA1547@gerg.ca>

On 15 February 2002, Patrick Callahan said:
> Hmm.  Tar is botched....  Could we learn anything from an attempt to 
> "redesign" tar's user interface?  Things are the way they are sometimes for 
> reasons we can't see. Or sometimes its just botched.  Either way, I'm 
> interested enough to discuss the fine points if you're up to it?

Perhaps "tar" isn't botched so much as it is a victim of its long
history and too many cooks stirring the broth.  Like many command-line
utilities, tar has both sub-commands and options.  There are two basic
ways to implement this: the CVS/Distutils model --
  cvs [global-opts] cmd [cmd-opts] [cmd-args]
  python setup.py [global-opts] cmd [cmd-opts]
-- or the tar/(pk)zip model:
  tar -cf foo.tar ...
  (pk)zip -a foo.zip ...

I happen to prefer the CVS/Distutils model, because it draws a clear
distinction between the command and options.  When the command is just
another option, it's never clear which options have to go where, which
ones are required, and what the relations between them are.  I claim
that CVS and a Distutils setup script have a clearer, easier-to-use
command-line UI than tar and (pk)zip.

tar is also confusing because you can get away without the hyphen:
"tar cf foo.tar ..." == "tar -cf foo.tar ...".  Not only am I confused
about which options are optional, I'm now confused about which arguments
are options.  Aiieeee!!  I'm sure this is a historical artifact with no
basis in good UI design.

> The tar utility has a lot of capabilities built into a single utility
> is that the source of the complexity?  Should tar have been split into
> multiple commands?

That's one possibility, and no doubt it was rejected in the early days
of Unix because memory and disk space were at a premium.  The semantic
difference between

  cvs update
  cvs diff

and

  cvs-update
  cvs-diff

is pretty minor, although it affects the implementation considerably.
(That interface would never fly for the Distutils, of course: the whole
point was to have a *single* setup script that does everything!)

        Greg
-- 
Greg Ward - nerd                                        gward@python.net
http://starship.python.net/~gward/
God made machine language; all the rest is the work of man.


From tim.one@comcast.net  Tue Feb 19 00:07:37 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 18 Feb 2002 19:07:37 -0500
Subject: [getopt-sig] Recent changes in Optik
In-Reply-To: <20020218000100.GA1006@gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEHENNAA.tim.one@comcast.net>

[Greg Ward]
> ...
> If you don't supply a type and one is needed, Optik uses "string"; if
> you don't supply a destination and one is needed, Optik extracts a
> destination from your first long option, or your first short option
> if none is supplied.
>
> For example, the following is now valid:
>
>   parser = OptionParser()
>   parser.add_option("-f", "--file")

+1

> ...
> The downside of this is that it enables programmers to be implicit
> rather than explicit, and we all know that explicit is better than
> implicit.

    parser.add_option("-f", "--file", dest="file")

isn't so much explict as pointlessly redundant.  "Implicit" would be
"options named 'file', or, more generally, starting with 'fi', are probably
names of files, so by default those will be stored as string values; otoh,
options with names starting with 'i' or 'n', or named 'count' or starting
with 'count', are probably integers, so by default get stored as integer
attributes, and unless integer conversion fails, in which case ...".

> Personally, I will probably take advantage of the default
> type a lot, but not the default destination.  YMMV.

So will yours <wink>:  I bet you start using the default destination a lot
too.  Having an option with a different name than the name the option's
value is stored under is a common cause of strange logic errors due to
confusion, so it's a Postive Good to let the default destination rule force
best practice.

Good show!



From gward@python.net  Tue Feb 19 02:43:57 2002
From: gward@python.net (Greg Ward)
Date: Mon, 18 Feb 2002 21:43:57 -0500
Subject: [getopt-sig] Recent changes in Optik
In-Reply-To: <20020218000100.GA1006@gerg.ca>
References: <20020218000100.GA1006@gerg.ca>
Message-ID: <20020219024357.GA5548@gerg.ca>

On 17 February 2002, I said:
> based on some of the feedback and requests seen in the past week, I've
> made some minor changes to Optik that, I think, will make it even easier
> to use.

Oh yeah, I forgot to mention the other changes I've been making.
Specifically, I've added an "examples" directory to the Optik CVS (will
be included in the next release) where I can play around with
implementing those features that people request but that I don't want to
add to Optik proper.  This forces me to actually test that Optik is
really as easily as extensible as I claim it is, and gives everyone else
canonical implementations of those heretical features.

So far, I've implemented a case-insensitive option parser and two
variations on "required options".  Surprise surprise, each new example
has resulted in minor tweaks to Optik -- specifically the OptionParser
constructor -- to make it easier to extend.  For example, with the code
currently in CVS, a case-insensitive option parser is pretty easy:

  class CaselessOptionParser (OptionParser):

      def _create_option_list (self):
          self.option_list = []
          self._short_opt = caseless_dict()
          self._long_opt = caseless_dict()
          self._long_opts = []
          self.defaults = {}

      def _match_long_opt (self, opt):
          return _match_abbrev(opt.lower(), self._long_opt.keys())

(The implementation of caseless_dict is left as an exercise to the
reader.  Or check out the current Optik CVS and see
examples/caseless.py.)

I invite everyone to take a check out the current Optik CVS and poke
around the examples directory.  If nothing else, this will serve as a
place to send people who ask for features that I don't think belong in
Optik proper.

        Greg
-- 
Greg Ward - Unix nerd                                   gward@python.net
http://starship.python.net/~gward/
"Question authority!"  "Oh yeah?  Says who?"


From a.t.hofkamp@tue.nl  Tue Feb 19 14:27:58 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Tue, 19 Feb 2002 15:27:58 +0100 (CET)
Subject: [getopt-sig] Argtools.py
Message-ID: <Pine.LNX.4.33.0202191507040.17149-100000@se-46.wpa.wtb.tue.nl>

Hello all,

I noticed a SIG about argument parsing, and since I have been working in that
area to create an open object oriented solution, here is my wood for the fire:

http://se.wtb.tue.nl/~hat/argtools

The source code is there, with an extremely simple example in it.
Also, the currently existing documentation, incomplete, and with spelling and
grammatical errors.
It is still lying on my desk to finish one day (the documentation, that is)
before I submit the module to the world.

Some key-points:
- It is fully object oriented. An option is an object, as well as the option
  parser. The typical use I have in mind is that a user derives a class from
  the parser he intends to use, which he then customizes for the options he
  wants.
  (sorry if my use of 'class' and 'object' is confusing, I am used to C++, with
  a much more static view on those concepts.)

- The toolkit is supposed to be open, in the sense that users should be able to
  change/add their own option (types) and/or parsers, depending on their needs.

- The class of an option (i.e. the option-type) decides what the option
  understands. There are a number of pre-defined options, yet a user is free to
  create new (derived) option-classes for his specific idiosyncratic options.
  For example, the 'StringListOption', which keeps a list of option-arguments,
  for example, if the optiontype associated with the name 'x' is of the class
  StringListOption, the command "-x spam -x eggs" will give the option the
  value ['spam','eggs'].

I'll download the SIG archive and the other packages for closer study shortly.

Albert
-- 
Constructing a computer program is like writing a painting



From shane@zope.com  Tue Feb 19 15:51:01 2002
From: shane@zope.com (Shane Hathaway)
Date: Tue, 19 Feb 2002 10:51:01 -0500
Subject: [getopt-sig] Yet another parser proposal
References: <68C679F3-2451-11D6-849E-0050E4A06334@laposte.net>
Message-ID: <3C727465.707@zope.com>

s.keim wrote:
> #  quickcmd.py
> #
> # a quick and dirty implementation around Shane Hathaway idea:
> # let's use python syntax for cmd line grammar specification.

This looks useful, but I'm surprised you went to so much work when a 
prototype was already in existence.  I thought we should first discuss 
whether this approach is a good one or not.

My goals for an option parser are as follows.  If anyone sees the goals 
differently, please help me out.

- Keep the options, the documentation, and the functionality in sync. 
I've written a few command line utilities, and over the life of a 
project sometimes an option is added or removed (or its meaning is 
changed) without updating the documentation.  Or even worse, the program 
might start ignoring an old option even though the option is still 
accepted and documented.

- Use a natural interface that doesn't have to be memorized or printed 
on a reference card.  Ideally, you only have to write a Python function 
or class.

- Generate a lot of the documentation automatically.

- Allow simple extensibility (groups of options, like "cvs -q up -dP") 
as well as complex extensibility (subclassing the option parser or the 
classes it uses).

The Python getopt module by itself doesn't meet any of these goals (I 
think).  It does meet lower-level goals such as parsing options in a 
consistent way, but it makes no attempt to keep options, documentation, 
and functionality in sync.  Its interface is cryptic (it accepts a 
structure of Python types with specially formatted strings, returning a 
different structure of Python types.)  It generates no documentation. 
It's flexible, but not really extensible.

As far as I can tell, Optik meets some of these goals.  It keeps the 
options and the documentation in sync, but it doesn't make an attempt to 
bind together the functionality and the options.  It has a simpler 
interface than getopt, but a programmer would still have to refer to a 
manual.  I don't know how well it generates documentation.  It appears 
to be very extensible, both in simple and complex ways.

The prototype ObjectCLI helps keep options and functionality in sync by 
deriving option specs from the function signature.  Most programmers 
remember to remove function parameters when they are no longer needed. 
ObjectCLI uses an interface Python programmers already know and use 
every day.  It generates a lot of the documentation, and it allows both 
simple and complex extensibility.

The ObjectCLI prototype does not currently make a full effort to keep 
options and documentation in sync, however.  And of course it currently 
assumes you always call a method of a Python class rather than a 
function.  s.keim's prototype appears to make the reverse assumption.

I'd also like to list some non-goals.  One non-goal is the level of 
complexity required to make a "tar" replacement without extending the 
parser.  I think we should assume if you're going to that extreme, you 
should have to extend the parser.  Another non-goal is a large number of 
options in a group.  Large numbers of options are hard for programmers 
to maintain and for users to understand.

So it appears to me that the concept behind ObjectCLI (and s.keim's 
code) meets these goals better than Optik does.  I would like to hear 
some discussion regarding whether you folks have the same goals and how 
you would evaluate the implementations.

Shane



From s.keim  Tue Feb 19 17:55:22 2002
From: s.keim (s.keim)
Date: Tue, 19 Feb 2002 18:55:22 +0100
Subject: [getopt-sig] Yet another parser proposal
In-Reply-To: <3C727465.707@zope.com>
Message-ID: <D88F94E8-2561-11D6-849E-0050E4A06334@laposte.net>

> This looks useful, but I'm surprised you went to so much work when a 
> prototype was already in existence.  I thought we should first discuss 
> whether this approach is a good one or not.
You are absolutely true! In fact I started saturday morning by writing 
some few lines just to see how it could look like... and discovered that 
the night had fallen down and that  I had spent all the day on this...
Not really reasonable ;-)

> - Keep the options, the documentation, and the functionality in sync.
+1

> - Use a natural interface that doesn't have to be memorized or printed 
> on a reference card.  Ideally, you only have to write a Python function 
> or class.
I believe that Python rock because it respects something that Frederick 
P. Brooks, in the Mythical Man-Month, call "conceptual integrity". For 
me that mean that the UI of a system should look for the minimal number 
of concepts and, probably more important avoid subtle variations  around 
theses concepts.
What I like in your proposal is that it reuse an existing protocol 
instead of creating a new one. The important points to check now are:
- is there better protocol to use?
- aren't we corrupting the class/function concept ?

> - Generate a lot of the documentation automatically.
I'm just thinking that maybe the parser documentation generation is 
quite related to the pydoc module.

> - Allow simple extensibility (groups of options, like "cvs -q up -dP") 
> as well as complex extensibility (subclassing the option parser or the 
> classes it uses).
We should be careful to not go to far in that way: in my opinion ease of 
use and reliability is far more important than extensibility. Because, 
it seems not really hard to parse the cmd line by hand, developers won't 
use the parser if they find it complex. So we should focus on the common 
needs and let exotic needs  be solved by externals tools.
If we don't take care, the danger is to fall in the hell of "there is 
more than one way to do it",  both for the UI of the script  and for the 
use of the parser ;-)

> I'd also like to list some non-goals.  One non-goal is the level of 
> complexity required to make a "tar" replacement without extending the 
> parser.  I think we should assume if you're going to that extreme, you 
> should have to extend the parser.  Another non-goal is a large number 
> of options in a group.  Large numbers of options are hard for 
> programmers to maintain and for users to understand.
+1

There is also two other points that need to be  discussed:

- the values storage: ObjectCli make it really elegant by using method 
arguments for value storage. I didn't made this choice because I planned 
that the function should only be used for analysis and not for 
processing. For Optik my feeling is that it do a little to much in this 
area by providing the action method, I think that only the default 
'store' action should be used and that it is the application job to 
decide what to do with the values.

- the type checking-conversion:  I find this feature interesting, but I 
wonder how far we should go on that way. And I definitively think that 
the type checking-conversion code should be reusable in other parts of 
the applications (because cmd line parsing is far to be the only place 
where you need to do this kind of job).
I'd like to militate for a simple protocol, that will allow to reuse 
theses objects for interactive text input checking for sample.



seb



From shane@zope.com  Tue Feb 19 18:51:43 2002
From: shane@zope.com (Shane Hathaway)
Date: Tue, 19 Feb 2002 13:51:43 -0500
Subject: [getopt-sig] Yet another parser proposal
References: <D88F94E8-2561-11D6-849E-0050E4A06334@laposte.net>
Message-ID: <3C729EBF.4060507@zope.com>

s.keim wrote:
>> - Use a natural interface that doesn't have to be memorized or printed 
>> on a reference card.  Ideally, you only have to write a Python 
>> function or class.
> 
> I believe that Python rock because it respects something that Frederick 
> P. Brooks, in the Mythical Man-Month, call "conceptual integrity". For 
> me that mean that the UI of a system should look for the minimal number 
> of concepts and, probably more important avoid subtle variations  around 
> theses concepts.

Right.  Reuse of user interfaces, if done properly, has numerous benefits.

> What I like in your proposal is that it reuse an existing protocol 
> instead of creating a new one. The important points to check now are:
> - is there better protocol to use?
> - aren't we corrupting the class/function concept ?

Personally I'm confident with this approach, since I'm very familiar 
with it from the Zope world, and because I've actually used it in a 
project.  Maintaining the option signature as part of the code is very 
programmer-friendly.

In Zope the class/function concept certainly wasn't corrupted IMHO. 
(Zope currently does have trouble with excessively large class 
hierarchies, but that is unrelated to this issue, and besides, the Zope 
3 project is seeking to solve the class hierarchy problem.)

>> - Generate a lot of the documentation automatically.
> 
> I'm just thinking that maybe the parser documentation generation is 
> quite related to the pydoc module.

True.  Does pydoc let you describe each function parameter in a 
machine-recognizable way?  That's one of the main features of javadoc. 
Parameter descriptions in docstrings would help us solve the issue of 
keeping the documentation in sync.

>> - Allow simple extensibility (groups of options, like "cvs -q up -dP") 
>> as well as complex extensibility (subclassing the option parser or the 
>> classes it uses).
> 
> We should be careful to not go to far in that way: in my opinion ease of 
> use and reliability is far more important than extensibility. Because, 
> it seems not really hard to parse the cmd line by hand, developers won't 
> use the parser if they find it complex. So we should focus on the common 
> needs and let exotic needs  be solved by externals tools.
> If we don't take care, the danger is to fall in the hell of "there is 
> more than one way to do it",  both for the UI of the script  and for the 
> use of the parser ;-)

Right.  I was thinking one might want to subclass the option parser to 
override the automatic documentation, and perhaps that's enough.

> There is also two other points that need to be  discussed:
> 
> - the values storage: ObjectCli make it really elegant by using method 
> arguments for value storage. I didn't made this choice because I planned 
> that the function should only be used for analysis and not for 
> processing. For Optik my feeling is that it do a little to much in this 
> area by providing the action method, I think that only the default 
> 'store' action should be used and that it is the application job to 
> decide what to do with the values.

Sounds right to me.

> - the type checking-conversion:  I find this feature interesting, but I 
> wonder how far we should go on that way. And I definitively think that 
> the type checking-conversion code should be reusable in other parts of 
> the applications (because cmd line parsing is far to be the only place 
> where you need to do this kind of job).
> I'd like to militate for a simple protocol, that will allow to reuse 
> theses objects for interactive text input checking for sample.

Well, I don't think we have to invent a "protocol", we just have to make 
the code modular and flexible.

I hope we'll see some comments from the rest of the group.  It feels as 
if you and I are in a different SIG entirely. ;-)

Maybe everyone needs to see a more concrete example of ObjectCLI.  Here 
is the code at the bottom of objectcli.py:


  class Program:
         "objectcli test program"

         def cmd1(self, alpha, b=0, *args):
             """
             Executes a command.

             --alpha=value  First argument
             -b             Enable second argument
             (Followed with arguments.)
             """
             print 'cmd1 got', alpha, b, args

         def cmd2(self, cat, d=0):
             """
             Executes a different command.

             --cat=value  First argument
             -d           Enable second argument
             """
             print 'cmd2 got', cat, d

     cli = ObjectCLI(Program())
     res = cli()
     sys.exit(res)


The only thing the programmer has to remember is to make an instance of 
the ObjectCLI class and call it.  The rest is simple Python.  Although 
ObjectCLI currently requires a class instance and a method name as the 
first argument, I now realize it should also allow you to pass a Python 
function, which would give you the equivalent of most command-line programs.

Shane



From gward@python.net  Wed Feb 20 02:24:36 2002
From: gward@python.net (Greg Ward)
Date: Tue, 19 Feb 2002 21:24:36 -0500
Subject: [getopt-sig] ObjectCLI
In-Reply-To: <3C6BD569.9030706@zope.com>
References: <3C6BD569.9030706@zope.com>
Message-ID: <20020220022436.GA8968@gerg.ca>

On 14 February 2002, Shane Hathaway said:
> ObjectCLI is a class that lets you invoke methods of a Python object 
> directly from the command line.  The concept is similar to Zope's 
> publisher, which lets you call methods directly from the Web.  Say you 
> created a class like this:

Hmmm.  I didn't really get this idea the first time I read your post.
I've been following the thread on this, and re-read your first post.  I
think I get it now, and I'm not too keen on it.  Let me explain why.

> class ServerControl:
>   """A proxy server."""
> 
>   def start(port):
>     """Starts the server on the specified port."""
> 
>   def stop():
>     """Stops the server."""
> 
> 
> With two or three lines of glue, you can then invoke the methods of a 
> ServerControl instance directly from the command line:
> 
> python server_control.py start --port=3128
> python server_control.py stop

Initially, this looks promising/interesting.  But what about programs
that don't have the "sub-command" paradigm?  How do you handle the
simple

   program --port=3128

case?  Also, what if I want to make -p and --port synonomous?  How do I
express that?  How would you handle standard, simple command-line
structures like

  ls -Fa /home/greg
  ls -l *.py
  cp -pr /tmp/foo /tmp/bar

?

> You can also get the documentation of ServerControl, primarily derived 
> from the docstrings:
> 
> python server_control.py --help

Interesting idea, but I worry about the overloading of docstrings.
IMHO, method/function docstrings are most useful for someone reading the
code: if I want to call function f, what arguments do I pass in, what
types must they be, what do they mean, how do they interact, what
happens to them, and what are the return values?  Those are very
different goals from the user of a program who wants to know what the
"start" sub-command does.

Also, where does the help for, eg., "--port" come from?  Are your method
docstrings structured so that help for individual options (method
arguments) can be pulled out?

> Major benefits:
> 
> - You almost never have to look up any library documentation to write 
> new command line utilities.  You just write a regular Python class.

I'm skeptical of that.  *You* almost never have to look up any
documentation, but you wrote it.  Likewise, I hardly ever have to look
at the Optik docs.  It's always obvious to the person who designed and
wrote it, and rarely to the poor schmo trying to use it.

> - The command line arguments and the implementation are less likely to 
> fall out of sync.

I'm not sure what you mean by "implementation".

> - The documentation comes directly from the docstrings.

Again, I'm not convinced this is a benefit.

Finally, you have alluded to a prototype implementation of ObjectCLI --
where?

        Greg
-- 
Greg Ward - Linux geek                                  gward@python.net
http://starship.python.net/~gward/
Vote Cthulhu -- why settle for a lesser evil?


From gward@python.net  Wed Feb 20 02:32:13 2002
From: gward@python.net (Greg Ward)
Date: Tue, 19 Feb 2002 21:32:13 -0500
Subject: [getopt-sig] DPyGetOpt
In-Reply-To: <B8933AF6.4E68%jonathan@onegoodidea.com>
References: <B8933AF6.4E68%jonathan@onegoodidea.com>
Message-ID: <20020220023213.GB8968@gerg.ca>

On 15 February 2002, Jonathan Hogg said:
> Just thought I'd drop a note to mention Bill Bumgarner's DPyGetOpt. As far
> as I'm aware, it has been out of active development for a long while.

Do you have a URL straight to the source?  I couldn't find it in a few
minutes of poking around ark.sourceforge.net.

More importantly, can you give some good reasons why we might consider
DPyGetOpt?  What does it offer that other option-parsing packages on the
table don't?

        Greg
-- 
Greg Ward - just another /P(erl|ython)/ hacker          gward@python.net
http://starship.python.net/~gward/
Time flies like an arrow; fruit flies like a banana.


From gward@python.net  Wed Feb 20 02:43:43 2002
From: gward@python.net (Greg Ward)
Date: Tue, 19 Feb 2002 21:43:43 -0500
Subject: [getopt-sig] Parse arguments according to a usage message?
In-Reply-To: <20020218151030.1E8CD2AA69@wireless-084-136.tele2.co.uk>
References: <20020218151030.1E8CD2AA69@wireless-084-136.tele2.co.uk>
Message-ID: <20020220024343.GC8968@gerg.ca>

On 18 February 2002, David Boddie said:
> I wonder whether there is any mileage in using part of the class that I
> mentioned in the "RFC: Option Parsing Libraries" thread in python-dev?
> 
> http://www-solar.mcs.st-and.ac.uk/~davidb/Software/Python/cmdsyntax/

Oooh, neat!  What a cool idea.  One thing to note is that you're
addressing the wider issue of *argument* parsing, whereas most people
are just interested in option parsing -- eg. Optik works by removing all
options and their arguments, and passing back a list of whatever's left
over.  This is a fairly common model, and it works for me.  But it
means every Optik-based script has code like

  [...]
  (options, args) = parser.parse_args()
  if len(args) != 3:
     parser.error("wrong number of arguments")

For a simple fixed number of positional arguments, this is not a big
deal, but it can be tiresome when you have a fairly complex syntax left
over after options have been parsed.  Your module, presumably, solves
this nicely by incorporating positional arguments into its domain.

There are also some fairly obvious limitations:

  * what about scripts with dozens of options?  the traditional
    "[-v] [-q] [-o outfile]" notation is great for 3 or 4 options,
    but loses its appeal pretty quickly.  (Try "man mpg123" for an
    example of what I mean.)

  * no strong typing -- everything's a string, and I have to explicitly
    convert numeric option arguments and catch the errors

  * how much say do I have in what happens to option arguments?  eg.
    is there an equivalent to Optik's "append" action, where each
    occurence of a particular option appends that occurence's argument
    to a list, rather than replacing previous values?

Must look at your code.  Neat idea.

        Greg
-- 
Greg Ward - geek-at-large                               gward@python.net
http://starship.python.net/~gward/
All the world's a stage and most of us are desperately unrehearsed.


From gward@python.net  Wed Feb 20 02:46:09 2002
From: gward@python.net (Greg Ward)
Date: Tue, 19 Feb 2002 21:46:09 -0500
Subject: [getopt-sig] Parse arguments according to a usage message?
In-Reply-To: <20020218151030.1E8CD2AA69@wireless-084-136.tele2.co.uk>
References: <20020218151030.1E8CD2AA69@wireless-084-136.tele2.co.uk>
Message-ID: <20020220024609.GD8968@gerg.ca>

On 18 February 2002, David Boddie said:
> I wonder whether there is any mileage in using part of the class that I
> mentioned in the "RFC: Option Parsing Libraries" thread in python-dev?
> 
> http://www-solar.mcs.st-and.ac.uk/~davidb/Software/Python/cmdsyntax/

Arghh!  I just noticed that your ZIP file has no version number, no
setup script, and no README file.  Please read
  http://python.org/doc/current/dist/dist.html
.  It's really not *that* hard.

grumble-whine-moan-bitch-complain...

        Greg
-- 
Greg Ward - Python bigot                                gward@python.net
http://starship.python.net/~gward/
I just read that 50% of the population has below median IQ!


From shane@zope.com  Wed Feb 20 05:45:29 2002
From: shane@zope.com (Shane Hathaway)
Date: Wed, 20 Feb 2002 00:45:29 -0500 (EST)
Subject: [getopt-sig] ObjectCLI
In-Reply-To: <20020220022436.GA8968@gerg.ca>
Message-ID: <Pine.LNX.4.33L2.0202200015090.5590-100000@localhost.localdomain>

On Tue, 19 Feb 2002, Greg Ward wrote:

> Initially, this looks promising/interesting.  But what about programs
> that don't have the "sub-command" paradigm?  How do you handle the
> simple
>
>    program --port=3128
>
> case?

It doesn't work in the current prototype, but instead of passing an
instance, you just pass a function.

>  Also, what if I want to make -p and --port synonomous?  How do I
> express that?

I'm sure we differ on this, but for me (and perhaps others), that is not a
goal.  You can still do it in a roundabout way, of course:

def program(p=0, port=0):
  if port == 0:
    port = p

>  How would you handle standard, simple command-line
> structures like
>
>   ls -Fa /home/greg
>   ls -l *.py

def ls(F=None, a=None, l=None, *filenames):
  ...

>   cp -pr /tmp/foo /tmp/bar

def cp(p=None, r=None, *filenames):
  ...

> > You can also get the documentation of ServerControl, primarily derived
> > from the docstrings:
> >
> > python server_control.py --help
>
> Interesting idea, but I worry about the overloading of docstrings.
> IMHO, method/function docstrings are most useful for someone reading the
> code: if I want to call function f, what arguments do I pass in, what
> types must they be, what do they mean, how do they interact, what
> happens to them, and what are the return values?  Those are very
> different goals from the user of a program who wants to know what the
> "start" sub-command does.

If you wrap a Python function rather than a Python instance (not possible
right now, but easy to add), I think the goals are quite similar.

> Also, where does the help for, eg., "--port" come from?  Are your method
> docstrings structured so that help for individual options (method
> arguments) can be pulled out?

No.  That is, admittedly, something I don't know how to solve yet.

> > Major benefits:
> >
> > - You almost never have to look up any library documentation to write
> > new command line utilities.  You just write a regular Python class.
>
> I'm skeptical of that.  *You* almost never have to look up any
> documentation, but you wrote it.  Likewise, I hardly ever have to look
> at the Optik docs.  It's always obvious to the person who designed and
> wrote it, and rarely to the poor schmo trying to use it.

The difference is there is only one entry point.  You only have to call
the framework.  The rest of the work is writing normal Python code.

> > - The command line arguments and the implementation are less likely to
> > fall out of sync.
>
> I'm not sure what you mean by "implementation".

I mean the code of the program.  With the code and the option signature
tightly bound, the code and the options are less likely to fall out of
sync.

> > - The documentation comes directly from the docstrings.
>
> Again, I'm not convinced this is a benefit.
>
> Finally, you have alluded to a prototype implementation of ObjectCLI --
> where?

In the "pma" subproject of the zodbex project at SourceForge.  See
zodbex.sf.net.

I'm pretty sure you and I have different goals.  I think your main goal is
to create a featureful option parser that does a better job than getopt.
My main goal is to eliminate the work of parsing options altogether,
possibly sacrificing some flexibility.  The differing goals result in very
different mindsets.  I'd like to know what other people's goals are.

Shane



From a.t.hofkamp@tue.nl  Wed Feb 20 08:58:03 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Wed, 20 Feb 2002 09:58:03 +0100 (CET)
Subject: [getopt-sig] Documentation, functionality, options, syncing
Message-ID: <Pine.LNX.4.33.0202200932580.18347-100000@se-46.wpa.wtb.tue.nl>

Hello,

Yesterday I read the mailing list archive, and did some thinking.
Having documentation automagically generated seems to be a hot topic, so let's
start with that one.

- I believe that it is not possible to get really good quality documentation
  from a number of one-liners that are part of an option. Especially, when
  talking about man-page size (or bigger).
  We should extend pydoc for such stuff.

  Even with lesser size, I'd like to be in control of the order in which
  help-text for options are printed (to allow grouping of similar options in
  the help), and have proper line-break routines for instance.

- I also believe that it is a mistake to learn new Pythonists that writing a
  one-liner for an option is documentation.
  I'd rather have no documentation (so it is glaring obvious there is none) than
  to have a one-liner that means nothing to anyone except the author.

- Please don't abuse pydoc strings for anything else than documentation of the
  implementation. Your option-objects/functions/etc may be easy enough for you
  not to write pydoc strings, but please don't make that decision for the rest
  of the world too.

- The argument of having to keep documentation and options in sync did not
  convince me that having help as part of the option, and having an option
  parser having to deal with help-output is good.

  On the other hand, I can understand the concern of keeping everything in
  sync. There fore, I'd like to propose two different approaches to the matter.
  Quite likely, others (with more experience in this field of keeping stuff in
  sync) can improve and/or propose entirely new approaches.

  * The first approach is to create the help text, while adding options to the
    parser, i.e. something like

    parser.AddOption('-h')
    helptext.append('-h get this helptext')

  * The first approach does not solve the problem if (like me) you want to be
    more in control. That could possibly be solved with checking afterwards:

    def usage():
       print "helptext how to use the program using -h, -x, and -t"
       parser.checkOptions(['-h','-x','-t'])

    i.e. I query the parser whether it understands -h, -x and -t. If it does
    not, or it has other options not mentioned here, it should protest.

  * Less strict checking is also possible:

    def usage():
       if parser.hasOption('-h'):
          print "help text about -h"
       if parser.hasOption('-x'):
          print "help text about -x"

    this also queries the parser, and uses its output to generate help. At the
    same time, it leaves the author in full control of how to generate the
    documentation.
    (e.g. one could load help from a file, and have multi-lingual help).


Albert
-- 
Constructing a computer program is like writing a painting



From a.t.hofkamp@tue.nl  Wed Feb 20 09:45:46 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Wed, 20 Feb 2002 10:45:46 +0100 (CET)
Subject: [getopt-sig] Looping versus object-oriented framework
Message-ID: <Pine.LNX.4.33.0202200959160.18347-100000@se-46.wpa.wtb.tue.nl>

Hello all,

A discussion that seems to have died out, but which I still consider relevant
is the looping versus object-oriented approach of option processing.

I feel there is a definite need for something extremely light-weight, like a
simple loop, as proposed by Russ Cox on 12 feb, like.

for opt in sys.argv[1:]:
   if opt == '-d':
      debug=1

As people correctly pointed out, this approach is broken due to complexities as
'-dc', which is either '-d -c' or '-d c' (with c the argument of the -d).

The step that we THUS need an object-oriented framework is a bit too big for
me. For simple scripts, as well as for people that do not grasp the
object-oriented concepts, the above is very appealing if it 'does the right
thing'. At least, the number of programs with broken option processing may
decrease, which is already a step forward.

On the other hand, for non-trivial option processing (or for people that very
much like the object-oriented techniques, like most of us), there is also a
need for more compact notation and/or more power of an object-oriented framework.

In short, I think there is a need for both.
Providing both gives power to those who need it and simplicity for those that
do not want power, with the option of 'upgrading' to the latter group as their
requirements increase.


I also think both approaches are not contradictionary with each other.
For the user they may seem contradictionary, but for us, programmers of the
option parsing module, the former is at the core of the object-oriented frame
work (someweher deep in the parser, we need to loop over the command-line,
exactly like the looping approach does).
Maintaining the loop approach is thus equivalent to maintaining the inner
parsing loop for us.
Releasing the looping interface to external users is then trivial.


Albert
-- 
Constructing a computer program is like writing a painting




From david@sleepydog.net  Wed Feb 20 12:27:02 2002
From: david@sleepydog.net (David Boddie)
Date: Wed, 20 Feb 2002 12:27:02 +0000
Subject: [getopt-sig] Parse arguments according to a usage message?
Message-ID: <20020220122850.C53752AA67@wireless-084-136.tele2.co.uk>

[Apologies to Greg: I meant to send this to the list as well, so he'll
get this message twice...]

On Wednesday 20 Feb 2002 2:43 am, Greg Ward wrote:
> On 18 February 2002, David Boddie said:
> > I wonder whether there is any mileage in using part of the class that I
> > mentioned in the "RFC: Option Parsing Libraries" thread in python-dev?
> >
> > http://www-solar.mcs.st-and.ac.uk/~davidb/Software/Python/cmdsyntax/
>
> Oooh, neat!  What a cool idea.  One thing to note is that you're
> addressing the wider issue of *argument* parsing, whereas most people
> are just interested in option parsing -- eg. Optik works by removing all
> options and their arguments, and passing back a list of whatever's left
> over.  This is a fairly common model, and it works for me.  But it
> means every Optik-based script has code like
>
>   [...]
>   (options, args) = parser.parse_args()
>   if len(args) != 3:
>      parser.error("wrong number of arguments")

I wrote two functions which I used to use for option finding in a command
line list, as from sys.argv[1:]. One would locate the option and any
following arguments, the other would remove what was found. Any left over
list entries were presumed to be arguments. This got very tiresome for
complicated command line syntaxes.

> For a simple fixed number of positional arguments, this is not a big
> deal, but it can be tiresome when you have a fairly complex syntax left
> over after options have been parsed.  Your module, presumably, solves
> this nicely by incorporating positional arguments into its domain.

It requires that all mandatory arguments (as opposed to options) are
given in order by the user. The parser can be asked to either look for
optional arguments in the order that they were specified, or out of
order as long as they don't disturb positional arguments.

For example:

infile [-l logfile] [-v] outfile

would allow

myfile -l log.txt -v output.txt
myfile -v -l log.txt output.txt

but not

myfile -v output.txt -l log.txt

Arbitrary numbers of trailing arguments are not supported at the moment,
although the matching algorithms could easily be modified to return
trailing arguments back to the caller.

> There are also some fairly obvious limitations:
>
>   * what about scripts with dozens of options?  the traditional
>     "[-v] [-q] [-o outfile]" notation is great for 3 or 4 options,
>     but loses its appeal pretty quickly.  (Try "man mpg123" for an
>     example of what I mean.)

I added the usual -abcde syntax which assumes that all the options are
single character options and that at least one is required. They can be
specified in any order, too. If you run the test.py script with a
syntax string containing something like this then you'll see what
-abcde is converted to. You may well be horrified. <wink>

The disadvantage to my assumption is that it allows someone to specify

-f foo

but can't cope with the user input

-ffoo

as it expects -f instead.

For lots of options and following arguments, then it's necessary to
specify them all in full. I don't really see a way around this other than
to format the syntax string nicely over several lines.

>   * no strong typing -- everything's a string, and I have to explicitly
>     convert numeric option arguments and catch the errors

That's why I suggested that it's only part of a solution. I don't mind
manually casting the strings to something useful, possibly using a
dedicated function to cope with raising exceptions or filling in default
values if the value given is invalid. Ideally, something would do this
for you, but I don't see the need for me to reinvent that mechanism when
Optik, for example, already provides that facility.

>   * how much say do I have in what happens to option arguments?  eg.
>     is there an equivalent to Optik's "append" action, where each
>     occurence of a particular option appends that occurence's argument
>     to a list, rather than replacing previous values?

Argument names (labels) should be unique within a given match of the
syntax definition. Note that

infile (--output-dir output)|(--output-file output)

contains the label "output" twice, but it will never be overwritten since
the user can't specify both the --output-dir and --output-file option as
the "|" character implies that one or the other must be given.

There's no facility for putting the values anywhere other than in a
dictionary at the moment.

> Must look at your code.  Neat idea.

Thanks. If you just run the source through pydoc, or look at the
dcoumentation at the URL I gave, it'll give you an idea of what the
parser supports.

David

________________________________________________________________________
This email has been scanned for all viruses by the MessageLabs SkyScan
service. For more information on a proactive anti-virus service working
around the clock, around the globe, visit http://www.messagelabs.com
________________________________________________________________________


From gward@python.net  Wed Feb 20 15:00:56 2002
From: gward@python.net (Greg Ward)
Date: Wed, 20 Feb 2002 10:00:56 -0500
Subject: [getopt-sig] ObjectCLI
In-Reply-To: <Pine.LNX.4.33L2.0202200015090.5590-100000@localhost.localdomain>
References: <20020220022436.GA8968@gerg.ca> <Pine.LNX.4.33L2.0202200015090.5590-100000@localhost.localdomain>
Message-ID: <20020220150056.GA9702@gerg.ca>

[Shane, on how to write programs that don't use the sub-command paradigm]
> It doesn't work in the current prototype, but instead of passing an
> instance, you just pass a function.

OK, that makes sense, and it's consistent with your answer to "How do
you implement ls?":

[me]
>   ls -Fa /home/greg
>   ls -l *.py

[Shane]
> def ls(F=None, a=None, l=None, *filenames):
>   ...

But it means *really* implement ls -- which on my system has 40
one-letter options and 37 long options, many of which are synonyms for
short options -- is impractical.

[me again]
> Also, what if I want to make -p and --port synonomous?  How do I
> express that?

[Shane]
> I'm sure we differ on this, but for me (and perhaps others), that is not a
> goal.

OK, then you're obviously uninterested in implementing something like
ls.  That's fine.

[Shane]
> I'm pretty sure you and I have different goals.  I think your main goal is
> to create a featureful option parser that does a better job than getopt.
> My main goal is to eliminate the work of parsing options altogether,
> possibly sacrificing some flexibility.  The differing goals result in very
> different mindsets.  I'd like to know what other people's goals are.

Right, I want to be able to implement the normal, sensible, conventional
GNU/Unix command-line syntax in every script I write with minimal fuss
and bother.  I want this mechanism to be flexible, extensible, and in
the standard library.  I don't think a solution that sacrifices
flexibility belongs in the standard library; things like having short
and long options that mean the same thing are just too important.
Strong typing and automatically generated help for each option are also
really nice.

More fundamentally, I think it's unrealistic to think that you can
eliminate the work of parsing options altogether.  Smells like wishful
thinking.  Anyways, I have convinced myself -- by writing Optik -- that
it's possible to make the work of option parsing very easy and
unobtrusive.  

        Greg
-- 
Greg Ward - Linux weenie                                gward@python.net
http://starship.python.net/~gward/
Jesus Saves -- and you can too, by redeeming these valuable coupons!


From shane@zope.com  Wed Feb 20 15:31:07 2002
From: shane@zope.com (Shane Hathaway)
Date: Wed, 20 Feb 2002 10:31:07 -0500
Subject: [getopt-sig] ObjectCLI
References: <20020220022436.GA8968@gerg.ca> <Pine.LNX.4.33L2.0202200015090.5590-100000@localhost.localdomain> <20020220150056.GA9702@gerg.ca>
Message-ID: <3C73C13B.4040106@zope.com>

Greg Ward wrote:
> [Shane]
> 
>>I'm pretty sure you and I have different goals.  I think your main goal is
>>to create a featureful option parser that does a better job than getopt.
>>My main goal is to eliminate the work of parsing options altogether,
>>possibly sacrificing some flexibility.  The differing goals result in very
>>different mindsets.  I'd like to know what other people's goals are.
> 
> Right, I want to be able to implement the normal, sensible, conventional
> GNU/Unix command-line syntax in every script I write with minimal fuss
> and bother.  I want this mechanism to be flexible, extensible, and in
> the standard library.  I don't think a solution that sacrifices
> flexibility belongs in the standard library; things like having short
> and long options that mean the same thing are just too important.
> Strong typing and automatically generated help for each option are also
> really nice.

Ok, we understand each other.  I think the difference boils down to the 
projects we're working on.  You're making utilities.  Utilities must 
have a very clean and often customized command-line interface.  "ls", 
"sort", "tar", etc. fall into this category.

I'm making applications.  Applications usually have a lot of features 
and making all the functions accessible via options is not a good UI. 
So applications often have subcommands, but the CLI is still an 
afterthought.  Most of the KDE apps (and probably GNOME apps as well) 
are examples of commands with a CLI as an afterthought.  "cvs" is an 
application with a reasonable UI.  "rpm" is also an application, though 
its option signature is peculiar.

I think there are a lot of Python applications that would benefit from a 
very easy way to provide a CLI.

> More fundamentally, I think it's unrealistic to think that you can
> eliminate the work of parsing options altogether.  Smells like wishful
> thinking.  Anyways, I have convinced myself -- by writing Optik -- that
> it's possible to make the work of option parsing very easy and
> unobtrusive.  

Yes, I agree it's wishful thinking to believe you can eliminate option 
parsing for general-purpose utilities.  But I think it is very 
reasonable for applications.

Shane



From gward@python.net  Thu Feb 21 02:41:47 2002
From: gward@python.net (Greg Ward)
Date: Wed, 20 Feb 2002 21:41:47 -0500
Subject: [getopt-sig] Documentation, functionality, options, syncing
In-Reply-To: <Pine.LNX.4.33.0202200932580.18347-100000@se-46.wpa.wtb.tue.nl>
References: <Pine.LNX.4.33.0202200932580.18347-100000@se-46.wpa.wtb.tue.nl>
Message-ID: <20020221024147.GA12322@gerg.ca>

On 20 February 2002, A.T. Hofkamp said:
> - I believe that it is not possible to get really good quality documentation
>   from a number of one-liners that are part of an option. Especially, when
>   talking about man-page size (or bigger).
>   We should extend pydoc for such stuff.

I completely agree.  What you get from "foo --help" is a brief summary
of what's available.  If you already know how foo works and what it
does, it's a useful memory aid.  If you're completely new to foo, it
might be useful to you -- or it might not.  There *must* be a man page
(or similar) to fall back on.  That's not a technical problem, that's a
social problem.

>   Even with lesser size, I'd like to be in control of the order in which
>   help-text for options are printed (to allow grouping of similar options in
>   the help), and have proper line-break routines for instance.

Optik prints the help in the same order you add options to your
OptionParser, so you have control over the order.  It also uses a good,
solid line-wrapping routine (distutils.fancy_getopt.wrap_text() -- which
will have to move if distutils.fancy_getopt is removed!) to generate
pretty good-looking output.

For example, here's a snippet of the "--help" output from a CD ripping
script I wrote; naturally, it uses Optik for the option-parsing:

usage: ripoff [options]

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -v                    increase verbosity level
  --verbose=V           set verbosity level to V
  -q, --quiet           run silently (verbosity = 0)
  -dDEVICE, --device=DEVICE
                        device file to open (default: /dev/cdrom on this
                        platform)
  -bDIR, --output-base=DIR
                        base output dir (default: /music/ogg)
  -tTRACKS, --tracks=TRACKS
                        tracks to rip (default: all) (example: 1,2-5,8)
  -p, --use-pipes, --synchronous
                        use named pipes for intermediate output (uses less
                        temporary disk space, works great if encoding is
                        faster than ripping) [default]
  -f, --use-files, --asynchronous
                        use real files for intermediate output (uses more
                        temporary disk space, but should be more reliable if
                        ripping is faster than encoding, and will free up your
                        CD-ROM drive sooner)
  [...]

Note that lines are nicely wrapped to 80 columns.  (Actually it's 78,
and yes it is hard-coded -- despite getting some useful answers on
c.l.py, I haven't gotten around to actually coding any of the 14
different ways to find out how big the terminal is.)  (You can see the
source for this script is at
  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ripoff/ripoff/ripoff?rev=1.42&content-type=text/vnd.viewcvs-markup
-- the option list is near the bottom, in main().)

The main thing missing from Optik is the ability to group options
thematically.  It's on my to-do list.

> - I also believe that it is a mistake to learn new Pythonists that writing a
>   one-liner for an option is documentation.  I'd rather have no
>   documentation (so it is glaring obvious there is none) than to have
>   a one-liner that means nothing to anyone except the author.

I quite disagree!  Any documentation is better than none at all.  And a
programmer who is incapable of writing decent per-option help -- ie. 95%
of programmers -- is probably also incapable of writing a decent man
page.

Again, this is a social problem, not a technical problem.  The Python
standard library can and should provide good tools, but it cannot turn
programmers into good documentors.

> - Please don't abuse pydoc strings for anything else than
>   documentation of the implementation.

+1

> - The argument of having to keep documentation and options in sync did not
>   convince me that having help as part of the option, and having an option
>   parser having to deal with help-output is good.
> 
>   On the other hand, I can understand the concern of keeping
>   everything in sync. There fore, I'd like to propose two different
>   approaches to the matter.  Quite likely, others (with more
>   experience in this field of keeping stuff in sync) can improve
>   and/or propose entirely new approaches.
> 
>   * The first approach is to create the help text, while adding options to the
>     parser, i.e. something like
> 
>     parser.AddOption('-h')
>     helptext.append('-h get this helptext')

If you're proposing a separate class whose sole responsibility is
formatting help, that's a good idea.  That's a large and hairy chunk of
my OptionParser class, and factoring it out to a new class might be a
good idea.  (Especially if we want to put in hooks for customizing help
output.)  But there's no good reason why we have to expose that
implementation detail to the programmer; what's wrong with spelling that

  parser.add_option('-h', help="get this help text")

anyways?  Under the hood, the OptionParser could be updating a
HelpFormatter, but that's irrelevant to the API.  Yes, it ties you to
having your help in the same order as your options are defined.  I think
that's a feature: it ties documentation clarity to code clarity; as one
increases (decreases), so does the other.  Can you think of an example
where you would want to define options in a different order than you
would document them?

        Greg
-- 
Greg Ward - geek-at-large                               gward@python.net
http://starship.python.net/~gward/
Don't hate yourself in the morning -- sleep till noon.


From gward@python.net  Thu Feb 21 02:49:44 2002
From: gward@python.net (Greg Ward)
Date: Wed, 20 Feb 2002 21:49:44 -0500
Subject: [getopt-sig] Looping versus object-oriented framework
In-Reply-To: <Pine.LNX.4.33.0202200959160.18347-100000@se-46.wpa.wtb.tue.nl>
References: <Pine.LNX.4.33.0202200959160.18347-100000@se-46.wpa.wtb.tue.nl>
Message-ID: <20020221024944.GB12322@gerg.ca>

On 20 February 2002, A.T. Hofkamp said:
> A discussion that seems to have died out, but which I still consider relevant
> is the looping versus object-oriented approach of option processing.

Definitely.  I prefer the OO style for most things, but if I had to
implement a tar/rpm/(pk)zip style interface, I think I'd prefer
something like Russ' iterator interface.  It just seems better suited to
cases where an earlier option affects the meaning and validity of later
options.  (I generally think this is bad UI design, which is why I
haven't felt the need for an iterator interface.)

[...digression on pros and cons of iterator interface...]

> In short, I think there is a need for both.  Providing both gives
> power to those who need it and simplicity for those that do not want
> power, with the option of 'upgrading' to the latter group as their
> requirements increase.

I have been thinking along those lines.  My plan is to see if I can
extract the bits of Optik's OptionParser class that grok "-" and "--"
from the bits that say "-a is valid, --flobnix is not" and bolt an
iterator interface on.  It will probably look a lot like Russ' code,
which I finally sat down and read tonight.  Then we could have one
module that provides both "OptionParser" (much like Optik's current
class) and "OptionIterator" (a stripped down option parser that lets the
application make all the tough decisions).

(I've delayed doing this because there are some other cleanups/tweaks I
want to make in Optik's code, and this is going to involve some fairly
heavy lifting.  Even though it wil be experimental and on a CVS branch,
I would like to make my life easier if it ever gets folded into the
trunk.)

> I also think both approaches are not contradictionary with each other.
> For the user they may seem contradictionary, but for us, programmers of the
> option parsing module, the former is at the core of the object-oriented frame
> work (someweher deep in the parser, we need to loop over the command-line,
> exactly like the looping approach does).

Absolutely!  It might still be possible to be all things to all
people... ;-)

        Greg
-- 
Greg Ward - Unix nerd                                   gward@python.net
http://starship.python.net/~gward/
NOBODY expects the Spanish Inquisition!


From a.t.hofkamp@tue.nl  Thu Feb 21 10:37:40 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Thu, 21 Feb 2002 11:37:40 +0100 (CET)
Subject: [getopt-sig] Looping versus object-oriented framework
In-Reply-To: <20020221024944.GB12322@gerg.ca>
Message-ID: <Pine.LNX.4.33.0202211018480.21287-100000@se-46.wpa.wtb.tue.nl>

On Wed, 20 Feb 2002, Greg Ward wrote:

> > I also think both approaches are not contradictionary with each other.
> > For the user they may seem contradictionary, but for us, programmers of the
> > option parsing module, the former is at the core of the object-oriented frame
> > work (someweher deep in the parser, we need to loop over the command-line,
> > exactly like the looping approach does).
>
> Absolutely!  It might still be possible to be all things to all
> people... ;-)

I think that is ESSENTIAL for a SIG.

If we are developing the standard Python option processing package, it should
work for every Python programmer in the world, preferably even those that today
do not dare using getopt.


Albert
-- 
Constructing a computer program is like writing a painting



From a.t.hofkamp@tue.nl  Thu Feb 21 13:56:16 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Thu, 21 Feb 2002 14:56:16 +0100 (CET)
Subject: [getopt-sig] option-class
Message-ID: <Pine.LNX.4.33.0202211138330.21287-100000@se-46.wpa.wtb.tue.nl>

Hello all,

Seeing all the positive reactions to Optik, I got curious how it compared to
'my' module 'argtools' (http://se.wtb.tue.nl/~hat/argtools).
Therefore, I took the example from the Optik page, and re-implemented it in
argtools.
Obviously, I am a bit biased to my own argtools package. However, please bear
with me. The real important part of this mail starts around point 4 and 5 when
I start discussing options and option classes.

The Optik example (from its hoem page):
-------------------------
from Optik import *

parser = OptionParser()
parser.add_option("-f", "--file",
                  action="store", type="string", dest="filename",
                  help="write report to FILE", metavar="FILE")
parser.add_option("-q", "--quiet",
                  action="store_false", dest="verbose", default=1,
                  help="don't print status messages to stdout")

(options, args) = parser.parse_args()
-------------------------
Since I have Python 1.5.2, I cannot run this example.

The argtools re-implementation:
-------------------------
from argtools import *

parser = OptionParser()

file    = parser.AddOption(StringOption("f file"))
verbose = parser.AddOption(  VoidOption("q quiet"))

parser.ProcessCommandLine()
args = parser.ArgList()
-------------------------
the results can be queried with

print "Results:"
print "File option: value="+file.Value()
print "Quiet option: used="+str(verbose.Used())



When comparing both pieces of code, I notice the following:

1) I can run the argtools example.
   One can argue that Python 2.0 is now minimal, but it is not standard
   everywhere. For example, my Python version came standard with Red Hat 7.1, a
   quite recent Linux distribution.
   We are also still using Red Hat 6.2 here, so world-wide there is still a
   large userbase with older versions of Python.
   Especially for less experienced users, requiring that the latest version
   must be installed just for an option processing package is a big requirement.
   (especially for a STANDARD PACKAGE).

2) Optik code looks bulkier.
   If I have more "-q"-like options, there is a lot redundancy, which leads to
   the usual maintenance and consistency problems.

   Part of the bulkiness originates from having to specify a helptext (argtools
   currently does not support help as part of the option), but if you remove
   that, like below
   -------------------------
   from Optik import *

   parser = OptionParser()
   parser.add_option("-f", "--file",
                     action="store", type="string", dest="filename")
   parser.add_option("-q", "--quiet",
                     action="store_false", dest="verbose", default=1)

   (options, args) = parser.parse_args()
   -------------------------
   it is still not pretty.
   There are exactly 80 chars on the -q line, exclusive the whitespace at the
   front, so having a single line for a single option inside a function or
   class is difficult at least.

3) Optik has at most 1 short and 1 long name for an option.
   This covers most uses. On the other hand, what if I have 3 names for a
   single option (or 2 long or short names). Some times, we don't have a choice
   in which or how many names should be supported.
   In argtools, I solved this by having a single string for all option names
   (with a short option being a single letter, and a long option being more
   than 1 letter).

4) Optik puts all results in 'options'. Argtools stores the results in the
   option itself.

   I think this is an important difference. Optik enforces the option
   processing results to a single place. In argtools, this can be decided by
   the programmer.
   I use as convention to derive a new class from the OptionParser, and store
   the options inside that class, but different solutions can be chosen too.
   For example, if options come from different parts, there may be no desire to
   have a central point where all results of all options are available.
   (What if 2 options use the same 'dest' ?!? (especially, if you have no
   control over those names)).

5) Extending Optik to new types of options is done by deriving a new option, a
   new parser, or by using callback functions.
   Argtools allows extensions with new types of options by deriving a new
   option class.
   I think that in a widely used standard Python package, the concept
   'callback' should not be used, and possibly, it should not even be available,
   as it defies all forms of good programming.
   Especially, in an object-oriented framework, deriving a new class is _THE_
   only way to go in my opinion. Anything else may be possible, but not for the
   normal user.

   Optik differentiates between adding a new type of option-argument (e.g. a
   boolean argument), and a new action that should take place. In the former
   case, a new option should be derived, in the latter case, a new parser
   should be derived.
   Also, I need to copy variables, make the right modifications in various
   places, etc. To me, extending Optik doesn't feel simple and logical.
   I don't see the logic behind the changes, other than the fact that the
   parser seems to be doing stuff it shouldn't do.

   Argtools on the other hand allows all extensions w.r.t. option-types or
   option processing of arguments to be done inside a (derived) option class.
   By limiting all changes to a single class, extending seems to be much easier.



Being the author of argtools, I am biased towards the good features of
argtools. However, I feel that the concept 'option' (or rather 'option-type')
as a seperate concept in the API has a lot of positive impact.
It makes the object-oriented nature of the framework more tangible, more explicit.
An option knows everything about itself. It knows its names, whether or not it
should take arguments, the syntax of those arguments, and what to do with the
argument, once it is encountered. For example, making an option that takes a
prime number is quite easy. Take the NumberOption, and derive an option with a
check that the number is a prime. Building an option that takes a complex
number is as simple as taking the ValueOption class, and adding a check whether
the argument is a proper complex number. Implementing an option that does
something different than storing the value is also done by deriving a new
option class.

With options knowing everything about themselves, the parser job is limited
to just parsing the command line, and offering the options to the list of
option objects, an extremely well-defined job.

All in all, the 'option' class concept that exists in argtools seems to make it
an extremely robust, extendible, and clean toolkit.


I hope I didn't offend people with my plugging of argtools, at least it was not
my intention.
My goal is not to push argtools as _THE_ solution. Instead, I hope to have
shown that adding the 'option class' concept in the final option processing
module which we are designing seems to make a lot of extensions fairly natural.
Therefore, the concept should be carefully considered for inclusion.


Albert
-- 
Constructing a computer program is like writing a painting



From a.t.hofkamp@tue.nl  Thu Feb 21 14:22:31 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Thu, 21 Feb 2002 15:22:31 +0100 (CET)
Subject: [getopt-sig] Documentation, functionality, options, syncing
In-Reply-To: <20020221024147.GA12322@gerg.ca>
Message-ID: <Pine.LNX.4.33.0202211506170.21287-100000@se-46.wpa.wtb.tue.nl>

On Wed, 20 Feb 2002, Greg Ward wrote:

> Note that lines are nicely wrapped to 80 columns.  (Actually it's 78,
> and yes it is hard-coded -- despite getting some useful answers on
> c.l.py, I haven't gotten around to actually coding any of the 14
> different ways to find out how big the terminal is.)  (You can see the

Shouldn't that be part of the line-wrapping library ?
>

> HelpFormatter, but that's irrelevant to the API.  Yes, it ties you to
> having your help in the same order as your options are defined.  I think
> that's a feature: it ties documentation clarity to code clarity; as one
> increases (decreases), so does the other.  Can you think of an example
> where you would want to define options in a different order than you
> would document them?

Ok, what about a multi-lingual program ?
I don't have much experience in this area, but I can imagine that the help of
an option should be available in multiple languages. All text in the program is
collected in 1 file for each language, and I have a unique key that finds the
text when I want to output a message.
Can I store the key as help text in the option ?

Another case is that I have written a frontend program that handles several
different types of files. The help text of the program does not only print the
options, but also the types of files it understands, with a description of what
it does with each type as a seperate piece of text. How does that fit in your
integrated help ?



I think the problem is not that it may not be useful to organize help text like
you do, but that it is an all-or-nothing option. If I want to use it, I need
the help to fit in the format/layout you designed. That may work tody, but what
about tomorrow ?
If I don't want to use the help texts, I must start from scratch, which is a
waste of time, since I probably could use some of the functionality.

I think it should be possible to give a bit more freedom, i.e. use a part of
it, and do the other stuff myself.
For example, allow querying the help text for (a list of) options (formatted or
non-formatted), thus allowing the use to write his own usage() (formatting)
code using information you manage.


Albert
-- 
Constructing a computer program is like writing a painting



From gward@python.net  Thu Feb 21 14:54:05 2002
From: gward@python.net (Greg Ward)
Date: Thu, 21 Feb 2002 09:54:05 -0500
Subject: [getopt-sig] option-class
In-Reply-To: <Pine.LNX.4.33.0202211138330.21287-100000@se-46.wpa.wtb.tue.nl>
References: <Pine.LNX.4.33.0202211138330.21287-100000@se-46.wpa.wtb.tue.nl>
Message-ID: <20020221145405.GA12793@gerg.ca>

[Albert -- could you please limit your posts to < 80 columns?
 Reformatting them is almost as annoying as trying reading them after
 "> " has been added.]

On 21 February 2002, A.T. Hofkamp said:
> Therefore, I took the example from the Optik page, and re-implemented it in
> argtools.
> Obviously, I am a bit biased to my own argtools package.

Excellent!  I've just started the process of reimplementing parts of
some of my real-life tools using other option parsing libraries.  I was
hoping a few other people would get into the game.

> The Optik example (from its hoem page):
[...]
> Since I have Python 1.5.2, I cannot run this example.

The goal of this SIG is to come up with a new and better option parsing
library for Python 2.3.  Therefore, compatibility with obsolete Python
versions that certain Linux distributors with their heads firmly stuck
in the sand keep distributing is a very minor issue, IMHO.

> The argtools re-implementation:
> -------------------------
> from argtools import *
> 
> parser = OptionParser()
> 
> file    = parser.AddOption(StringOption("f file"))
> verbose = parser.AddOption(  VoidOption("q quiet"))
> 
> parser.ProcessCommandLine()
> args = parser.ArgList()
> -------------------------
> the results can be queried with
> 
> print "Results:"
> print "File option: value="+file.Value()
> print "Quiet option: used="+str(verbose.Used())
> 
> 
> 
> When comparing both pieces of code, I notice the following:
> 
> 1) I can run the argtools example.
>    One can argue that Python 2.0 is now minimal, but it is not standard
>    everywhere.

Again, this is mostly irrelevant since we're trying to come up with a
new getopt for Python 2.3.  If you're stuck with Python 1.5.2, you
certainly won't have the new getopt from 2.3.

> 2) Optik code looks bulkier.

That has improved a bit in the current Optik CVS; the -f/--file option
from the Optik home page can now be spelled:

  parser.add_option("-f", "--file", dest="filename",
                    help="write report to FILE", metavar="FILE")

...and if you don't mind having a filename stored in a variable called
'file', you can drop the 'dest' and 'metavar' arguments too.

>    if I have more "-q"-like options, there is a lot redundancy, which
>    leads to the usual maintenance and consistency problems.

Boolean options are tricky.  Very often, I want two different options
writing to the same variable; the canonical example is -v sets 'verbose'
true and -q sets it false.  You can do this with Optik, but it isn't
terribly pretty:

  parser.add_option("-v", action="store_true", dest="verbose")
  parser.add_option("-q", action="store_false", dest="verbose")

That looks OK, but it does get bulky when you add synynoms (--verbose,
--quiet), help, and a default value.  I'd like to hear how other option
libraries handle this very common case!

>    There are exactly 80 chars on the -q line, exclusive the whitespace at the
>    front, so having a single line for a single option inside a function or
>    class is difficult at least.

I find most option specifications take 2 or 3 lines of code.  That's too
bad, but you get a lot of bang for your buck.  So I'm not hung up on
cramming each option spec into a single line.

> 3) Optik has at most 1 short and 1 long name for an option.

Incorrect.  You can have as many short and long names as you like:

  parser.add_option("-p", "--use-pipes", "--synchronous", ...)

>    In argtools, I solved this by having a single string for all option names
>    (with a short option being a single letter, and a long option being more
>    than 1 letter).

Yuck.  Why put multiple values into a single string when something like

  def add_option (self, *args)

works just as well?  This is quite possibly my favourite trick in Optik;
see OptionParser.add_option() and Option.__init__() in
  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/optik/optik/lib/option_parser.py?rev=1.33&content-type=text/vnd.viewcvs-markup
  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/optik/optik/lib/option.py?rev=1.15&content-type=text/vnd.viewcvs-markup
for how it works.

> 4) Optik puts all results in 'options'. Argtools stores the results in the
>    option itself.

Right: input in one object, output in another.  I don't like the idea of
commingling option specifications (which are fixed by the application
programmer) with option values (which are chosen by the user, vary from
run to run, and are completely transient).  It just feels wrong.  With
the Optik way, you can easily reuse the same Option object for multiple
arg parsing runs; can you do that with Argtools?

Also, if you have 17 different options, it's quite easy to pass them all
around together: just pass the options object.  When you put each option
value into a separate local variable, what then?  You'll probably create
your own options object just to keep them together; why not let the
option parsing library do that for you?

However, I should play with Argtools a bit to see how I like it first.

>    (What if 2 options use the same 'dest' ?!?o

Often a desirable feature, as in -v/-q above.  But it does increase the
semantic load on the programmer.

> (especially, if you have no
> control over those names)).

These are variable names.  Why would you not have control over the
variable names used in your program?

> 5) Extending Optik to new types of options is done by deriving a new option, a
>    new parser, or by using callback functions.
>    Argtools allows extensions with new types of options by deriving a new
>    option class.
>    I think that in a widely used standard Python package, the concept
>    'callback' should not be used, and possibly, it should not even be available,
>    as it defies all forms of good programming.

I'm not keen on callbacks, but sometimes it's easier to write a callback
than to derive your own Option subclass.  An application with a lot of
callbacks would probably be better off with an iterative interface anyways.

>    Especially, in an object-oriented framework, deriving a new class
>    is _THE_ only way to go in my opinion. Anything else may be
>    possible, but not for the normal user.

That really strikes me as overly dogmatic.  Subclassing is a great
extensibility mechanism, and in an OO framework it's almost always the
preferred extensibility mechanism.  But (heresy alert!) there's more
than one way to do it.

>    Optik differentiates between adding a new type of option-argument (e.g. a
>    boolean argument), and a new action that should take place.

Exactly.  That's the key insight that took me *years* to arrive at; when
I had it, I sat down and wrote Optik within a week.  (I have been
pondering this problem for a looong time, and I could never get around
the idea that "list to append to" is a type just like "string" or "int"
is a type.  Thinking in that rut makes it awfully hard to, say, add an
"output directory" type, and then automagically gain the ability to
accumulate a list of output directories.  When I finally figured out
that "append to list" is an *action* like "store to a variable",
everything just came together.)

>    In the former
>    case, a new option should be derived, in the latter case, a new parser
>    should be derived.

Incorrect.  In both cases, you extend the Option class.  See
extending.txt:
  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/~checkout~/optik/optik/extending.txt

>    Also, I need to copy variables, make the right modifications in various
>    places, etc. To me, extending Optik doesn't feel simple and logical.

It's a bit tricky because you have to set various class attributes in
your subclass.  It's either that or define a bunch of methods to
substitute for the class attributes -- eg. "is_typed_action()" rather
than TYPED_ACTIONS -- and then subclasses would have to override those
methods instead.  

>    I don't see the logic behind the changes, other than the fact that the
>    parser seems to be doing stuff it shouldn't do.

Have you read extending.txt?  And what precisely do you mean by "the
parser doing stuff it shouldn't do"?

>    Argtools on the other hand allows all extensions w.r.t. option-types or
>    option processing of arguments to be done inside a (derived) option class.
>    By limiting all changes to a single class, extending seems to be much easier.

OK, I want to add an "output directory" option type.  Can I get
"accumulate a list of output directories" for free?  Or do I have to add
a "list of output directories" option type too?

> I hope I didn't offend people with my plugging of argtools, at least
> it was not my intention.  My goal is not to push argtools as _THE_
> solution. Instead, I hope to have shown that adding the 'option class'
> concept in the final option processing module which we are designing
> seems to make a lot of extensions fairly natural.  Therefore, the
> concept should be carefully considered for inclusion.

I'm not offended, whatever you might think after reading this post.
Looks like there are definitely some good ideas in Argtools, and I'll
give it a good look.

        Greg
-- 
Greg Ward - Linux nerd                                  gward@python.net
http://starship.python.net/~gward/
Don't hate yourself in the morning -- sleep till noon.


From gward@python.net  Thu Feb 21 15:05:13 2002
From: gward@python.net (Greg Ward)
Date: Thu, 21 Feb 2002 10:05:13 -0500
Subject: [getopt-sig] Documentation, functionality, options, syncing
In-Reply-To: <Pine.LNX.4.33.0202211506170.21287-100000@se-46.wpa.wtb.tue.nl>
References: <20020221024147.GA12322@gerg.ca> <Pine.LNX.4.33.0202211506170.21287-100000@se-46.wpa.wtb.tue.nl>
Message-ID: <20020221150513.GB12793@gerg.ca>

[me, on line-wrapping in Optik's help output]
> Note that lines are nicely wrapped to 80 columns.  (Actually it's 78,
> and yes it is hard-coded -- despite getting some useful answers on
> c.l.py, I haven't gotten around to actually coding any of the 14
> different ways to find out how big the terminal is.)  (You can see the
 
[Albert]
> Shouldn't that be part of the line-wrapping library ?

What line-wrapping library?  Currently all there is is the wrap_text()
function buried in the distutils.fancy_getopt module.  This probably
should be moved to somewhere more visible in the standard library, but I
haven't dealt with that yet.


> Ok, what about a multi-lingual program ?  I don't have much experience
> in this area, but I can imagine that the help of an option should be
> available in multiple languages. All text in the program is collected
> in 1 file for each language, and I have a unique key that finds the
> text when I want to output a message.  Can I store the key as help
> text in the option ?

Nobody said that help text has to be a static string.

   parser.add_option("-f", "--file",
                     help=lookup_text("OPT_HELP_FILE"))

Or if you prefer implicit over explicit:

   parser.add_option("-f", "--file",
                     help=_("Read input from FILE"))

(I believe this is how Mailman does I18N: the _() function uses one
language -- presumably English -- as the key, and looks up the same
message in a different language.)

If you want to get a bit fancier:

   parser.add_option("-f", "--file",
                     metavar=_("FILE"),
                     help=_("Read %s" % _("FILE")))

so your help output might read either
  -fFILE, --file=FILE        Read FILE
or
  -fFICHIER, --file=FICHIER  Lire FICHIER
(Sorry, English and French are all I know.)

But this is purely guesswork.  I know very little about I18N.
                       
> Another case is that I have written a frontend program that handles
> several different types of files. The help text of the program does
> not only print the options, but also the types of files it
> understands, with a description of what it does with each type as a
> seperate piece of text. How does that fit in your integrated help ?

Again, nobody said help text has to be static.

> I think the problem is not that it may not be useful to organize help
> text like you do, but that it is an all-or-nothing option. If I want
> to use it, I need the help to fit in the format/layout you
> designed. That may work tody, but what about tomorrow ?  If I don't
> want to use the help texts, I must start from scratch, which is a
> waste of time, since I probably could use some of the functionality.

Certainly Optik's help-formatting code needs to be split up into
multiple methods to make it more customizable and extensible.  Possibly
it needs to be factored out into a separate class.  But it's hard to do
that without real-life examples of how help formatting needs to be
customized or extended.

        Greg
-- 
Greg Ward - geek-at-large                               gward@python.net
http://starship.python.net/~gward/
Know thyself.  If you need help, call the CIA.


From s.keim  Fri Feb 22 09:52:55 2002
From: s.keim (s.keim)
Date: Fri, 22 Feb 2002 10:52:55 +0100
Subject: [getopt-sig] Extracting data formating from option parser
In-Reply-To: <20020221150513.GB12793@gerg.ca>
Message-ID: <F1FC6C54-2779-11D6-A7A4-0050E4A06334@laposte.net>

I have already expressed in the list the idea to let the parser delegate 
string conversion and type checking to an helper object..
I would try to explain it more precisely.

Even if ,for the moment, I'm only interested in the string->value 
conversion, in the next part of the document I will call Formatter the 
object which has the responsibility of data conversion and type checking.
This is highly inspired from the NSFormatter class of the 
NextStep/MacOSX framework.
http://developer.apple.com/techpubs/macosx/Cocoa/Reference/Foundation/Java/
Classes/NSFormatter.html

String evaluation is a very common programming task you need it for 
sample:
- for interactive user interface
- text files processing
- command line management ;)
...
So Python is quite well featured in this area, mainly because 'type' 
functions (int, float, complex ...) can be used with a string parameter 
and return ValueError for invalid format.

I don't think that there is a really great need for more so we could 
define, for the input part, a Formatter as:
- an object callable with a string argument
- that return the value of the result or raise a ValueError

Some minors improvements could be defined:
* FormatValueError: inherit from ValueError and contain the index 
position of the error in the string
And, not related to option parsing but interesting for use in 
interactive UI:
* verify(xx) check an unfinished string for correctness
* complete(xxx) auto-completion features

An then, instead of hard-coding the conversion stuff in the parser, we 
could reuse predefined Formatter classes:
parser.add_option("-d",  "--date",  type=DateFormater ("%y/%m/%d") )

One argument against this idea is that the current Optik solution allow 
to share informations from the parser to the data conversion mechanism. 
To know if this is a rebuttal problem, I'd like to have some example of 
situation where you need this kind of informations from the parser for 
data conversion.

I think that the major strength of my proposal is that it is generic, so 
I would finish with an example of how  this could be used in gui 
programming:
date_form = DateFormater ("%y/%m/%d")
date_text = Text(dialog, formater=date_form)

and then you could write code like:
date_text.value = time.time()
new_time = date_text.value
without boring anymore with string conversions.

What are you opinions about this?
Do you think it would be interesting if I work on a more constructed 
proposal,  or  is it a definitively bad idea?



From gward@python.net  Sat Feb 23 20:49:15 2002
From: gward@python.net (Greg Ward)
Date: Sat, 23 Feb 2002 15:49:15 -0500
Subject: [getopt-sig] Extracting data formating from option parser
In-Reply-To: <F1FC6C54-2779-11D6-A7A4-0050E4A06334@laposte.net>
References: <20020221150513.GB12793@gerg.ca> <F1FC6C54-2779-11D6-A7A4-0050E4A06334@laposte.net>
Message-ID: <20020223204915.GA18726@gerg.ca>

On 22 February 2002, s. keim said:
> I have already expressed in the list the idea to let the parser delegate 
> string conversion and type checking to an helper object..

I don't see why such a strong separation of duties is needed.  In Optik,
string conversion and type checking are the same: if something can be
converted from a string to the type-of-choice, then it is type-checked.
If not, it is of the wrong type.  

> Even if ,for the moment, I'm only interested in the string->value 
> conversion, in the next part of the document I will call Formatter the 
> object which has the responsibility of data conversion and type checking.

Now you're talking about "data conversion" in addition to "string->value
conversion" and "type checking".  I'm usually quite keen on separation
of duties, but I really don't see the distinction between these three
tasks in the context of command-line argument parsing.

> This is highly inspired from the NSFormatter class of the 
> NextStep/MacOSX framework.
> http://developer.apple.com/techpubs/macosx/Cocoa/Reference/Foundation/Java/
> Classes/NSFormatter.html
> 
> String evaluation is a very common programming task you need it for 
> sample:
> - for interactive user interface
> - text files processing
> - command line management ;)

Ahh.  I think you are suffering from a case of premature
overgeneralization syndrome.  A command-line parsing library should be
geared specifically towards parsing command-line arguments.  It should
not be designed with the hypothetical, "maybe someday" idea that would
could reuse some of it in a GUI toolkit.  If you do that, you'll make
the command line library so arcane and overgeneralized that nobody will
be able to understand it (either for use or maintenance), and it will
*still* have some deep, fundamental assumptions that in the end will
make it a bad fit for GUI programming.  Don't go there.  Keep it simple.

It's pretty rich for me, of all people, to be preaching this, but: make
it as simple as you can and still have it work.  Good rule to follow.

> An then, instead of hard-coding the conversion stuff in the parser, we 
> could reuse predefined Formatter classes:
> parser.add_option("-d",  "--date",  type=DateFormater ("%y/%m/%d") )

Please take a good hard look at the Optik source code, specifically the
Option class in option.py.  It doesn't achieve quite this level of
flexibility, but I didn't view that as a requirement.  But it gets
pretty far with the notion of "type-checking" functions.  All you need
to do to define a new type is write a function that takes a string
value, makes sure it can be converted to your type, and returns a value
of that type.  If your "type" is something like "readable file", then
you might just return the original string unchanged, and the sole
purpose of the type-checking function is type-checking.  Or it might
return a file object -- whatever.

> What are you opinions about this?
> Do you think it would be interesting if I work on a more constructed 
> proposal,  or  is it a definitively bad idea?

Two words: premature overgeneralization!

        Greg
-- 
Greg Ward - just another Python hacker                  gward@python.net
http://starship.python.net/~gward/
Question authority!


From wolfson@midway.uchicago.edu  Sat Feb 23 22:20:56 2002
From: wolfson@midway.uchicago.edu (Ben Wolfson)
Date: Sat, 23 Feb 2002 16:20:56 -0600 (CST)
Subject: [getopt-sig] Parse arguments according to a usage message?
In-Reply-To: <20020220024343.GC8968@gerg.ca>
Message-ID: <Pine.GSO.4.21.0202231547340.23655-100000@harper.uchicago.edu>

On Tue, 19 Feb 2002, Greg Ward wrote:

[cmdsyntax.py]

>There are also some fairly obvious limitations:
>
>  * what about scripts with dozens of options?  the traditional
>    "[-v] [-q] [-o outfile]" notation is great for 3 or 4 options,
>    but loses its appeal pretty quickly.  (Try "man mpg123" for an
>    example of what I mean.)
>
>  * no strong typing -- everything's a string, and I have to explicitly
>    convert numeric option arguments and catch the errors
>
>  * how much say do I have in what happens to option arguments?  eg.
>    is there an equivalent to Optik's "append" action, where each
>    occurence of a particular option appends that occurence's argument
>    to a list, rather than replacing previous values?

It seems to me that you could get around these objections by adapting some
of Optik's ideas.

Instead instantiating a parser with a vanilla usage string, first
instantiate a bunch of instances of an option class, and then pass them,
along with plain strings, to the parser.  Something like:

outfile = Option('-o', '--output', type='string', action="store")
verbose = Option('-v')
s = Syntax("[-v] infile [-o outfile]", outfile, verbose)

The Syntax class would use the instances to find out what the options are
called, and match them up with the syntax string passed in.  For that
matter, for an uncomplicated command-line syntax, Syntax could infer
what the string must be from the Option instances passed in; for mpg123
(assuming the Options have already been instantiated) it could just be

synt = Syntax(None, t,s,c,v,q,y .... )

Or something like that.  That wouldn't notate that mpg123's -o option can
only have three values ([ -o s | -o h | -o l ])  but it would be easy (I
assume) to subclass Optik's Option class to indicate that a given option
can only take certain options, and then communicate that to the Syntax
class.

For not-too-complicated interfaces, though, you would have to type out the
entire interface string, and when it's not too complicated, it's not a
whole lot different from Optik anyway.  But I still think there's potential
here.

-- 
BTR    
Back in the day, the women would talk about Michelangelo.
 -- CRGRE




From smurf@noris.de  Sun Feb 24 13:21:13 2002
From: smurf@noris.de (Matthias Urlichs)
Date: Sun, 24 Feb 2002 14:21:13 +0100
Subject: [getopt-sig] Option package requirements
Message-ID: <20020224142113.E26294@noris.de>

Having browsed the archives a bit, I'm now going to put my foot in the
door (probably while somebody else tries to close it ;-) and chime in
with my own option package requirements:

- Modules should be able to add their own options without requiring
  modification of the main program (so there should be a single default
  option-parser object which is instanciated only once).

- Options and help strings should be declared ONCE, in ONE place.
  If that means extra work for the option parser when it has to actually
  display the help text, so be it.

That pretty much rules out anything but Optick at this point. ;-)


To justify the following more-or-less esoteric requirements (which
arguably should go to the Optick list :-): 

IMHO it's of no use at all to specify an option processing package which
ends up being unable to handle real-world requirements. Real programs have
a whole bunch of usages, and I want my Python program to be able to
emulate those other programs.

- Positional vs. global options. Suppose I want to rewrite
  sox, the command-line sound converter. It has this Usage string:
  
  sox [ gopts ] [ fopts ] ifile [ fopts ] ofile [ effect [ effopts ] ]

  This might boil down to an Optik call like
  
  parser.add_option(1, ...)

  meaning that this "option" is processed for the first non-option argument.
  (In this case, a callback function would stash the input file's fopts
  someplace safe so that they're not overwritten by the output file's
  fopts.)

- A way to end option processing / gobble up all remaining arguments.

  Suppose I want to rewrite ssh. It haas this option:

  ssh HOST foo ...

  which calls the remote "foo" program with any options given afterwards.
  (Enclosing these options in a string, like "sh -c" wants me to, is NOT
  the Right Way.)

  Note: This example would add the end-processing option to the 'option'
  2. Other programs use "-e" or "--exec" for this. Same thing, if
  the above option is implemented.
  
- Freely mix options and arguments.

  See any GNU Makefile; they end up calling "makeinfo FOO.texi -o FOO.info".

- Shorthand arguments.

  Suppose I want to rewrite gpg. It allows me to shorten long option names
  as long as they're unambiguous, so "--output-file" can be written
  "--output-f".

- Optional option arguments.

  Suppose I want to rewrite the GCC front-end. See, for example, its "-g"
  and "-O" options.

  gcc file       -- no -O given
  gcc -O file    -- -O without value
  gcc -O1 file   -- -O with value
  gcc -O 1 file  -- process the two files "1" and "file".

  Long options should work likewise:

  gcc --opt file    -- -O without value
  gcc --opt= file   -- -O with empty value (impossible with short options)
  gcc --opt=1 file   -- -O with value

  Another example for this is the mysql client program (the "-p" option).

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/
-- 
No experiment is ever a complete failure, inasmuch as a well-written
account of it can serve admirably as a bad example.


From rsc@plan9.bell-labs.com  Sun Feb 24 14:07:04 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Sun, 24 Feb 2002 09:07:04 -0500
Subject: [getopt-sig] Option package requirements
Message-ID: <561f89cc0e768008b18cfae9469e6948@plan9.bell-labs.com>

> IMHO it's of no use at all to specify an option processing package which
> ends up being unable to handle real-world requirements. Real programs have
> a whole bunch of usages, and I want my Python program to be able to
> emulate those other programs.

All of the examples you propose have non-standard behaviors,
introduce ambiguities, and only serve to confuse users.  There's
no good justification for any of the examples you present.
In _my_ HO, all of them are examples of bad design, and it
would not be such a bad thing if the standard Python option
parser made it very hard or impossible to emulate them.

Russ



From smurf@noris.de  Sun Feb 24 15:14:53 2002
From: smurf@noris.de (Matthias Urlichs)
Date: Sun, 24 Feb 2002 16:14:53 +0100
Subject: [getopt-sig] Option package requirements
In-Reply-To: <561f89cc0e768008b18cfae9469e6948@plan9.bell-labs.com>; from rsc@plan9.bell-labs.com on Sun, Feb 24, 2002 at 09:07:04AM -0500
References: <561f89cc0e768008b18cfae9469e6948@plan9.bell-labs.com>
Message-ID: <20020224161452.G26294@noris.de>

Hi,

Russ Cox:
> In _my_ HO, all of them are examples of bad design, and it

I am _not_ disagreeing here.

> would not be such a bad thing if the standard Python option
> parser made it very hard or impossible to emulate them.
> 
Personally I'd settle for "add some dire warnings to the documentation".

My point is that none of these options are in any way difficult to add
if the standard parser either provides for them as-is, or is structured so
that it's easy to override.

For instance, optional arguments can be done by allowing hooks for an
option-specific parse function which is allowed to mangle the argument
list; the positional non-argument stuff is easy if you allow the empty
string as an option value (which should not accept a singe "-" argument,
as that is used to specify stdin/out in some programs).

Hooks like these would allow for special requirements, while still not
explicitly supporting them, thus (at least) requiring the programmer to
THINK before implementing nonstandard stuff.

But the requirement to occasionally do non-standard processing definitely
exists, and so IMHO it should be supportable.

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/
-- 
Don't believe everything you hear or anything you say.


From gward@python.net  Sun Feb 24 21:43:07 2002
From: gward@python.net (Greg Ward)
Date: Sun, 24 Feb 2002 16:43:07 -0500
Subject: [getopt-sig] Option package requirements
In-Reply-To: <561f89cc0e768008b18cfae9469e6948@plan9.bell-labs.com>
References: <561f89cc0e768008b18cfae9469e6948@plan9.bell-labs.com>
Message-ID: <20020224214307.GA19197@gerg.ca>

[Russ Cox reacts to Matthias Ulrich's proposals]
> All of the examples you propose have non-standard behaviors,
> introduce ambiguities, and only serve to confuse users.  There's
> no good justification for any of the examples you present.
> In _my_ HO, all of them are examples of bad design, and it
> would not be such a bad thing if the standard Python option
> parser made it very hard or impossible to emulate them.

I completely agree with you here.  (And, for the record, I think sox
definitely belongs in the doghouse with tar and rpm for unnecessarily
difficult command-line UIs.  sox is the worst offender in my books, but
that's probably only because I have long since made my peace with tar
and rpm (and I use Debian now ;-).)

But, irony of ironies, it seems to me that your iterator interface is
exactly the tool for people who *do* want to perpetrate such awkward,
non-standard UIs.  A one-option-at-a-time, let-me-write-all-the-code-
dammit interface is *just* what the doctor ordered if you want to deal
with anti-social things like context-sensitive options, multiple sets of
options on the same command line, order sensitivity, and the like.
Writing such a beast with Optik would be really painful -- you'd need a
lot of callbacks that interact with each other in complicated ways.  I
think that discouraging that sort of bad UI is a *feature*.  I thought
that the appeal of your iterator interface is that it exposes the
flexibility to be evil for people who (think they) need it.

        Greg
-- 
Greg Ward - Linux nerd                                  gward@python.net
http://starship.python.net/~gward/
Never put off till tomorrow what you can put off till the day after tomorrow.


From rsc@plan9.bell-labs.com  Sun Feb 24 21:53:23 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Sun, 24 Feb 2002 16:53:23 -0500
Subject: [getopt-sig] Option package requirements
Message-ID: <6b30363a5df2147bdb3ee0fee459248f@plan9.bell-labs.com>

> But, irony of ironies, it seems to me that your iterator interface is
> exactly the tool for people who *do* want to perpetrate such awkward,
> non-standard UIs.  A one-option-at-a-time, let-me-write-all-the-code-

Not really.  You couldn't write tar or sox with it.
(I don't know enough about rpm's syntax to judge.)

Once it hits a non-option argument it stops.
That kills off sox and makeinfo and cc.
In the case of tar, the first - is optional.
Can't do that.  In the -O1 case, you can't have
optional arguments.

I don't remember the other examples well
enough, but I'm fairly sure that none of them
are possible with my interface (unless you dig
around in the internals).

So I think the iterator discourages that sort
of UI just as much as an object-based parser.

Russ



From gward@python.net  Sun Feb 24 21:55:50 2002
From: gward@python.net (Greg Ward)
Date: Sun, 24 Feb 2002 16:55:50 -0500
Subject: [getopt-sig] Option package requirements
In-Reply-To: <20020224142113.E26294@noris.de>
References: <20020224142113.E26294@noris.de>
Message-ID: <20020224215550.GB19197@gerg.ca>

On 24 February 2002, Matthias Urlichs said:
> - Modules should be able to add their own options without requiring
>   modification of the main program (so there should be a single default
>   option-parser object which is instanciated only once).
> 
> - Options and help strings should be declared ONCE, in ONE place.
>   If that means extra work for the option parser when it has to actually
>   display the help text, so be it.

I'm +1 on both of those, in case there was any doubt.  ;-)

> IMHO it's of no use at all to specify an option processing package which
> ends up being unable to handle real-world requirements. Real programs have
> a whole bunch of usages, and I want my Python program to be able to
> emulate those other programs.

I'm strongly opposed to further complicating Optik in order to support
esoteric needs and downright bad UI design.  My goal right now is to
keep refactoring Optik so that esoteric needs can be met by subclassing,
and to simplify it to answer the (fair) criticisms that it's too
complex.

Regarding your specific proposals...

> - Positional vs. global options. Suppose I want to rewrite
>   sox, the command-line sound converter. It has this Usage string:
>   
>   sox [ gopts ] [ fopts ] ifile [ fopts ] ofile [ effect [ effopts ] ]
> 
>   This might boil down to an Optik call like
>   
>   parser.add_option(1, ...)
> 
>   meaning that this "option" is processed for the first non-option argument.
>   (In this case, a callback function would stash the input file's fopts
>   someplace safe so that they're not overwritten by the output file's
>   fopts.)

Positional arguments are not options.  Options are not positional
arguments.  Confusing the two will lead to a confused interface and
deeply confused users.  (As an experienced programmer with a good grasp
of command-line UI who has struggled with sox to get it to do what I
want, I can testify that at least one user is deeply confused by it.)

Anyways, you could achieve this with Optik by setting
allow_interspered_args false, and restarting argument parsing multiple
times.  Eg.

  parser1 = OptionParser(...)
  parser1.disable_interspersed_args()
  (options1, args) = parser1.parse_args()
  ifile = args.pop(0)
  parser2 = OptionParser(...)   # partially different option list this time
  parser2.disable_interspersed_args()
  (options2, args) = parser2.parse_args(args)
  ofile = args.pop(0)
  if args:
      effect = args.pop(0)
      # ... validate effect ...
      parser3 = OptionParser(...)   # completely different option list
      (options3, args) = parser3.parse_args(args)
      # ... bomb if any args left ...

It's not terribly pretty, but it should work.

> - A way to end option processing / gobble up all remaining arguments.
> 
>   Suppose I want to rewrite ssh. It haas this option:
> 
>   ssh HOST foo ...
> 
>   which calls the remote "foo" program with any options given afterwards.
>   (Enclosing these options in a string, like "sh -c" wants me to, is NOT
>   the Right Way.)

Again, you can do this with Optik now by setting
enable_interspersed_args false.  Optik will stop parsing when it hits
the HOST argument, and then you can do what you wish with the remaining
arguments.

> - Freely mix options and arguments.

Optik already handles this by default, but you can turn it off when you
need to (see above).

> - Shorthand arguments.

Optik already handles this.  If for some perverse reason you do *not*
want option abbreviation (eg. you hate your users and you want them to
suffer), you'll be able to disable it in Optik 1.3 by overriding one
method of OptionParser.

> - Optional option arguments.
> 
>   Suppose I want to rewrite the GCC front-end. See, for example, its "-g"
>   and "-O" options.
> 
>   gcc file       -- no -O given
>   gcc -O file    -- -O without value
>   gcc -O1 file   -- -O with value
>   gcc -O 1 file  -- process the two files "1" and "file".

Yet another argument for optional option arguments.  I'm glad we're all
using the same awkward terminology for this concept now, because it *is*
awkward.  Currently not supported directly by Optik, but you should be
able to pull it off with a callback.  I'm going to try to implement it
through subclassing to see if I can refactor that code to be more
friendly, but I'm skeptical it will work.  I think this case is rare
enough that just saying, "Use a callback" is good enough.  (Especially
if there's a canonical implementation in the examples/ directory.)

        Greg
-- 
Greg Ward - programmer-at-large                         gward@python.net
http://starship.python.net/~gward/
Quick!!  Act as if nothing has happened!


From gward@python.net  Mon Feb 25 02:51:35 2002
From: gward@python.net (Greg Ward)
Date: Sun, 24 Feb 2002 21:51:35 -0500
Subject: [getopt-sig] Comparing option libraries
Message-ID: <20020225025135.GA997@gerg.ca>

Hi all --

I finally sat down this afternoon and evening and re-implemented the
same command-line interface with three different option libraries:
Optik, Russ Cox' ArgParser class, and Albert Hofkamp's argtools.  Rather
than bore you all with the details here, I've put up a web page with the
results:

  http://optik.sourceforge.net/compare.html

(This will move to the getopt-sig's page as soon as I have access to the
machine where SIG pages are maintained.)

Right now, I'd particularly like Russ and Albert to comment on my use of
their libraries.  Once we have disposed with the unfairness inherent in
my knowing how to use Optik better than the other two, I'll post my
opinions of the process.  In the meantime, everyone please take a look
at the code!  I think I'll go add some comments now...

        Greg
-- 
Greg Ward - Linux weenie                                gward@python.net
http://starship.python.net/~gward/
In space, no one can hear you fart.


From rsc@plan9.bell-labs.com  Mon Feb 25 19:47:05 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Mon, 25 Feb 2002 14:47:05 -0500
Subject: [getopt-sig] Comparing option libraries
Message-ID: <38ad5ef69b913f591a528e012082fa00@plan9.bell-labs.com>

Have you significantly changed the Optik library
since this discussion started?  If I'd seen something
like http://optik.sourceforge.net/ripoff_optik.py
instead of the example at http://optik.sourceforge.net/,
I wouldn't have made such a big deal about the iterator.

What if Optik defined two subclasses, one that takes
no arguments and one that takes one argument?
And why not make type a real type instead of a string?
Further, what if the action string could be something
to eval instead of needing to be a canned action string?

Then we can remove most of the noise:

option_list = [
	Helptext("\nVerbosity options:"),
	Optflag("-v", action="verbose += 1",
		help="increase verbosity level"),
	Optarg("--verbose", var="V", type=int,
		help="verbosity level"),
	Optflag("-q", "--quiet", action="verbose = 0",
		help="run silently (verbosity=0)"),

	Helptext("\nRipping options:"),
	Optarg("-d", "--device",
		help="device file to open (default: %s on this platform)"
			% cdrom.get_default_device()),
	Optarg("-b", "--output-base", var="DIR",
		help="base output directory"),
	Optarg("-t", "--tracks", 
		help="tracks to rip (default: all) (example: 1,2-5,8)"),
	Optflag("-p", "--use-pipes", "--synchronous",
		help="use named pipes for intermediate output "
			"(uses less temporary disk space, works great "
			"if encoding is faster than ripping) [default]"),
	Optflag("-f", "--use-files", "--asynchronous", action="use_pipes = 0",
		help="use real files for intermediate output (uses more "
			"temporary disk space, but should be more reliable "
			"if ripping is faster than encoding, and will free "
			"up your CD-ROM drive sooner)"),
	Optflag("-k", "--keep-tmp",
		help="put temporary (.wav) files in output dir, and don't "
			"delete them when done with them (implies -f)"),
	Optflag("-R", "--rip-only",
		help="just do digital audio extraction -- no encoding "
			"(implies -k, -f)"),
	Optflag("-P", "--playback",
		help="play encoded audio files as encoding proceeds"),
	Optflag("-e", "--eject",
		help="eject CD when finished"),

	Helptext("\nTagging options:"),
	Optarg("-A", "--artist",
		help="use ARTIST for tagging and file naming "
			"(overrides CDDB/CDindex)"),
	Optarg("-L", "--album",
		help="use ALBUM for tagging and file naming"),
	Optarg("-D", "--disc", type=int,
		help="set disc number to DISC (for multi-disc sets)"),
	Optarg("-y", "--year",
		help="use YEAR for tagging"),
	Optarg("--offset", type=int, default=0,
		help="add OFFSET to track numbers for tagging and file naming"),
]

Finally, there should be some similar way to specify how to parse the
rest of the command line; otherwise you're encouraging people
to use options when they're not really necessary, because
they get the nice benefits of things like type checking.

Russ



From gward@python.net  Mon Feb 25 23:02:59 2002
From: gward@python.net (Greg Ward)
Date: Mon, 25 Feb 2002 18:02:59 -0500
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <38ad5ef69b913f591a528e012082fa00@plan9.bell-labs.com>
References: <38ad5ef69b913f591a528e012082fa00@plan9.bell-labs.com>
Message-ID: <20020225230259.GA4654@gerg.ca>

On 25 February 2002, Russ Cox said:
> Have you significantly changed the Optik library
> since this discussion started?

The main changes are that you can now often get away without explicitly
supplying a 'dest', and 'type' defaults to "string".  This definitely
reduces the noise level, and cost about 3 lines of code to add.

> If I'd seen something
> like http://optik.sourceforge.net/ripoff_optik.py
> instead of the example at http://optik.sourceforge.net/,
> I wouldn't have made such a big deal about the iterator.

And if you hadn't made such a big deal about the iterator, I wouldn't
have been driven to make these improvements in Optik.  So obviously it
was all part of my master plan to force myself into improving Optik just
a little bit more.  ;-)

> What if Optik defined two subclasses, one that takes
> no arguments and one that takes one argument?

Hmmmm.  I presume you're referring to Optflag and Optarg:

> 	Optarg("-b", "--output-base", var="DIR",
> 		help="base output directory"),
> 	Optarg("-t", "--tracks", 
> 		help="tracks to rip (default: all) (example: 1,2-5,8)"),
> 	Optflag("-p", "--use-pipes", "--synchronous",
> 		help="use named pipes for intermediate output "
> 			"(uses less temporary disk space, works great "
> 			"if encoding is faster than ripping) [default]"),

First, I think I would call those two classes FlagOption and
ValueOption, which is a bit more verbose, but a bit more verbose.
(>wink<).  And I would add add_flag_option() and add_value_option() to
OptionParser, since I still like having two ways to populate the option
list (even if that violates Python's Prime Directive).

Second, my now-standard response to anyone proposing a new feature for
Optik is this: implement it yourself and show me where Optik needs
refactoring.  Usually the challenge goes unanswered, and I end up doing
it myself, which is just as good.  Maybe better, because then I have a
good understanding of how it needs to be refactored.  Anyways, you have
my blessing, nay encouragement, to implement these subclasses (including
an OptionParser with add_{flag,value}_option()) and to howl loudly if
some part of Optik needs refactoring to support them.

> And why not make type a real type instead of a string?

I think what you want is to be able to say (eg.)

   add_option("-f", type=int, ...)

instead of 

   add_option("-f", type="int", ...)

-- correct?  So what you're really asking for is that Optik type's be
specified by identifiers, not by strings.  There happen to be convenient
built-in identifiers for the three types Optik supports by default.  But
if you want to define a type for, say, "readable file", then you also
have to supply an identifier to represent that type.  My guess is you'll
create a global string variable somewhere and users of your new type
will just have to import that variable from somewhere, which is slightly
*less* convenient than using a string.  But it is also slightly less
error-prone than supplying a string, since Python itself will tell you
you've typed 'fiel' instead of 'file', whereas currently only Optik
tells you this.  Hmmm.

Tell you what: I think I'll make Optik recognize the str, int, and float
builtins as synonyms for "string", "int", and "float".  Then if you
write your own Option subclass that adds new types, you can specify
those types however you like.

> Further, what if the action string could be something
> to eval instead of needing to be a canned action string?

Hmmmm.  I'm not keen on overloading "action" to mean "one of these
canned actions, OR some bit of code to eval" -- makes it tricky to know
what to do if someone mistypes an action string.

Two alternatives:

  * add another Option attribute (and kw arg to Option.__init__() and
    OptionParser.add_option()) where you specify a bit of code to
    eval, eg.

      Optflag("-v", eval="verbose += 1",
              help="increase verbosity level")

    Supplying this would implicitly set 'action' to "eval" (a new canned
    action).

  * overload the callback mechanism, so you can supply a bit of code
    to evaluate instead of a function.

I prefer the former -- I think actions should do one thing and do them
well, and I'm not shy about adding new actions.
          
> Finally, there should be some similar way to specify how to parse the
> rest of the command line; otherwise you're encouraging people
> to use options when they're not really necessary, because
> they get the nice benefits of things like type checking.

I've always thought that the only thing to be done with positional args
is make sure there are the right number of them, eg. you supply a pair
(n, m) and Optik makes sure that n <= len(args) <= m.  But
"type-checking" positional args makes sense if there are "types" like
"readable file" or "output directory".

Hmmmm.  I'm open to interface suggestions.  Ideas, anyone?

        Greg
-- 
Greg Ward - programmer-at-big                           gward@python.net
http://starship.python.net/~gward/
Drive defensively -- buy a tank.


From gward@python.net  Mon Feb 25 23:50:01 2002
From: gward@python.net (Greg Ward)
Date: Mon, 25 Feb 2002 18:50:01 -0500
Subject: [getopt-sig] We have a web page.
Message-ID: <20020225235001.GA5077@gerg.ca>

I finally got around to creating a web page for this SIG.
It's at
  http://www.python.org/sigs/getopt-sig/

The page of comparison implementations I posted last night has moved
from Optik's page to the getopt-sig's page, which I think is more
appropriate:
  http://www.python.org/sigs/getopt-sig/compare.html

Enjoy --

        Greg
-- 
Greg Ward - Unix nerd                                   gward@python.net
http://starship.python.net/~gward/
Think honk if you're a telepath.


From rsc@plan9.bell-labs.com  Tue Feb 26 02:33:39 2002
From: rsc@plan9.bell-labs.com (Russ Cox)
Date: Mon, 25 Feb 2002 21:33:39 -0500
Subject: [getopt-sig] Option package requirements
Message-ID: <0d1f8ae39b185ce0eedf6d7736a5761f@plan9.bell-labs.com>

> But the requirement to occasionally do non-standard processing definitely
> exists, and so IMHO it should be supportable.

This is where we differ.  I admit that some programs
may want to do non-standard processing, but I believe
we should be discouraging this, and one of the best
ways is not to make any explicit attempts to support it.
It's not like using the standard option parser will be
the only way to parse the command-line.  If someone wants
a non-standard argument syntax, they can write their own
parser.  That's good discouragement.

Russ


From jlh@cox.net  Tue Feb 26 04:28:14 2002
From: jlh@cox.net (Jeff Hinrichs)
Date: Mon, 25 Feb 2002 22:28:14 -0600
Subject: [getopt-sig] Hidden Options?
Message-ID: <2e0501c1be7e$019f5320$6702a8c0@gato>

wrt Optik:
I don't believe that it is currently possible to create a hidden option.
(i.e. --debug) where the parser would know about it but not talk about it.
That is to say, it wouldn't be listed in the --help listing.

Would this be of value to anyone else?

-Jeff Hinrichs
jlh@cox.net




From goodger@users.sourceforge.net  Tue Feb 26 04:44:04 2002
From: goodger@users.sourceforge.net (David Goodger)
Date: Mon, 25 Feb 2002 23:44:04 -0500
Subject: [getopt-sig] Hidden Options?
In-Reply-To: <2e0501c1be7e$019f5320$6702a8c0@gato>
Message-ID: <B8A07CC3.1F272%goodger@users.sourceforge.net>

Jeff Hinrichs wrote:
> wrt Optik:
> I don't believe that it is currently possible to create a hidden option.
> (i.e. --debug) where the parser would know about it but not talk about it.

It is; see advanced.txt:

    To omit an option entirely, use the special value optik.SUPPRESS_HELP.

-- 
David Goodger    goodger@users.sourceforge.net    Open-source projects:
 - Python Docstring Processing System: http://docstring.sourceforge.net
 - reStructuredText: http://structuredtext.sourceforge.net
 - The Go Tools Project: http://gotools.sourceforge.net



From jlh@cox.net  Tue Feb 26 05:23:11 2002
From: jlh@cox.net (Jeff Hinrichs)
Date: Mon, 25 Feb 2002 23:23:11 -0600
Subject: [getopt-sig] Hidden Options?
References: <B8A07CC3.1F272%goodger@users.sourceforge.net>
Message-ID: <001501c1be85$aee18e70$6702a8c0@gato>

David,
Thanks for the quick response.  As I, <open mouth, insert manual>

-Jeff
p.s.
"Optik Rocks!"

----- Original Message -----
From: "David Goodger" <goodger@users.sourceforge.net>
To: "Jeff Hinrichs" <jlh@cox.net>; <getopt-sig@python.org>
Sent: Monday, February 25, 2002 10:44 PM
Subject: Re: [getopt-sig] Hidden Options?


> Jeff Hinrichs wrote:
> > wrt Optik:
> > I don't believe that it is currently possible to create a hidden option.
> > (i.e. --debug) where the parser would know about it but not talk about
it.
>
> It is; see advanced.txt:
>
>     To omit an option entirely, use the special value optik.SUPPRESS_HELP.
>
> --
> David Goodger    goodger@users.sourceforge.net    Open-source projects:
>  - Python Docstring Processing System: http://docstring.sourceforge.net
>  - reStructuredText: http://structuredtext.sourceforge.net
>  - The Go Tools Project: http://gotools.sourceforge.net
>
>



From a.t.hofkamp@tue.nl  Tue Feb 26 10:58:59 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Tue, 26 Feb 2002 11:58:59 +0100 (CET)
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <20020225025135.GA997@gerg.ca>
Message-ID: <Pine.LNX.4.33.0202260959320.29337-100000@se-46.wpa.wtb.tue.nl>

On Sun, 24 Feb 2002, Greg Ward wrote:

> Hi all --

Hi Greg,

I think you are doing a terrific job of answering all issues raised in this
SIG, many thanks for that.


Now, with respect to the re-implementation of the command-line interface of
ripoff, using argtools:
(details are on http://www.python.org/sigs/getopt-sig/compare.html).

In the argtools-version of the source, you raise a number of questions of 'how
to do this in argtools' ?
The answer to those questions is both low tech (i.e. 'this is how'), and
conceptual (i.e. 'this is why it is not easily possible').

Unfortunately, I have little time this week for the first part, so I will
concentrate on the second part (which is also more interesting here imho).

Argtools is built on the following assumption:

An option object (i.e. an instance of an option class) represents a single
command-line concept.
Example, the 'verbosity of the program' (i.e. what level of output should the
program produce) is a single concept, which should be implemented by a single
option object.

>From this assumption follows that options are independant from each other
(since they are related to different concepts), and that the order of options
is not relevant (also because they control different concepts).

I think the above are sound design principles. It should be non-trivial to
deviate from them.


Now on to your questions:
- How to implement the --file and --pipe options ?
  From your comment I understand that these options are mutually exclusive,
  i.e. the user cannot specify both.
  In Optik, both options set the same value (one does something like
  'opions.output='file', and the other does 'options.output='pipe').

  It is true that it is not easy to implement in argtools in the sense that you
  can do:
      self.file = self.AddOption(VoidOption('file'))
      self.pipe = self.AddOption(VoidOption('pipe'))
  because these are 2 seperate option objects.
  Due to the above assumption, this means in argtools that you have a concept
  'output to file y/n', as well as a concept 'output to pipe y/n'.

  The current ripoff program implements the policy that 'the last setting is
  valid', i.e. if a user specifies 'ripoff --file --pipe', it is considered to
  be equivalent to 'ripoff --pipe'.
  I disagree with this equivalence notion. If a user specified both options, it
  is likely that he wants both options.
  By ignoring some of his options, you create unexpected behaviour of the program.
  You should at least warn the user that you ignore some part of the
  commandline, I personally would not even execute the command until the user
  makes an unambigious choice.

  If the implementor of a program really wants this behaviour, then at least he
  should make that decision rather than the option processing module making
  that decision for him.
  That is, he should specify somewhere what to do when the user specifies both
  --pipe and --file.
  I think argtools behaves properly by making the implementor at least realize
  that the ambiguity exists, and that he should deal with it.


  So, can --pipe and --file be implemented in argtools ?
  Yes.
  Since we are talking about a single option concept, there should be a single
  option object for the option parser.
  You would derive an option class from BaseValueOption, which recognizes
  'file' and 'pipe', and sets the value of the option to e.g. 'file'/'pipe'.
  Very rough, untested implementation:

  class RipoffOutputOption(BaseValueOption):
     def _init__(self,defaultval='pipe'):
        BaseValueOption.__init__(self,'not used',defaultval)
     def Match(self,opt,arg):
        if opt=='file' or opt=='pipe':
	   self.SetUsed(1)
	   self.SetSet(1)
	   self.SetValue(opt)
	   return _OPTIONMATCH
        return _NOMATH

  Now you have a single option object that understands both --pipe and --file,
  and handles them, as in the ripoff program.

  I think it is better to throw the --file and --pipe options out, and define
  --output=file and --output=pipe. This is implemented in the EnumOption of
  argtools. No ambiguities, no user confusion.

  Having to create a derived class for this case is good. It is a sufficient
  hurdle for people to reconsider their option interface before implementing
  something confusing as --file and --pipe.

- A similar story holds for the verbosity of the output, except that the
  complexity of the realtions between the various option settings is even more
  complex.

- A counting option (i.e. an option object that counts how many times an option
  like -v has been specified) is currently not implemented in argtools (though
  it should be!!).
  An implementation:

  class CountingOption(BaseValueOption):
      def __init__(self,optnames,defaultval=0):
         BaseValueOption.__init__(self,optnames,defaultval)

      def Match(self,opt,arg):
         if self.Parse(opt):
             self.SetUsed(1)
             self.SetSet(1)
             self.SetValue( self.Value()+1 )
             return argtools._OPTIONMATCH
         return argtools._NOMATCH

  Then, -v can be specified as "self.v = self.AddOption(CountingOption('v'))".

  Note that in my opinion, the above code should be considered part of
  argtools, not of the ripoff program (and thus not being counted as 'lines of
  code' for the ripoff implementation).

- Your line count is not entirely fair due to some interfacing problems.
  If you would develop ripoff using argtools, then you would _NOT_ make copies
  from the parser object to the options object, you would use the parser object
  itself for querying options.
  Alternatively, you would add the option objects into 'options' rather than in
  the parser object.
  In both cases, the lines of code for copying option values should be excluded
  from the line-count.

- For options with arguments, how to see whether the option is used, etc ?
  argtools has 3 methods for querying an option object with a value:
  - Used(): returns 1 if the option is specified, 0 otherwise
  - Set(): returns 1 if the option on the command line has an argument
  - Value(): the value of the argument of the option object.

  Thus:
  if not option.Used():
     # option is not specified
  elif option.Used() and not option.Set():
     # option without argument is specified, for example --log
  elif option.Used() and option.Set():
     # option with argument is specified, for example --log=$HOME/mylog
     # value of the argument in option.Value()

  Obviously, if --log is not allowed, then the second case never happens, and
  Set() and Used() are always the same.

  Notice the strict seperation of usage and value. In many other option
  toolkits including Optik, these notions are mixed together into special
  values of the argument.

  While this seems to work for most option values, it does give some special
  care when handling the value of an option.
  (in a dynamic language such as Python, the problem is much less severe, but
  for example in C++, how would you differentiate between nothing, --bool,
  --bool=false, and --bool=true ?)

  I think the seperation between usage and value is good. It gives less
  surprises to the programmer of what to check.


So the questions are:

- Is the idea of 'an option object represents a single concept' good ?

and

- Should we have a seperation between usage and value of an option ?

(for me, both questions should be answered 'yes').

Albert
PS With your comparison page, you suggest some form of competition. I do not
   have that intention. Optik is much more world-ready than argtools. On the other
   hand, Optik does make some assumptions that may not always be correct/wanted.
   Argtools seems to make it more difficult for the programmer in those cases.

   By introducing (some of the) concepts from argtools in the final package of
   this SIG, everybody wins.
-- 
Constructing a computer program is like writing a painting




From smurf@noris.de  Tue Feb 26 12:37:14 2002
From: smurf@noris.de (Matthias Urlichs)
Date: Tue, 26 Feb 2002 13:37:14 +0100
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <Pine.LNX.4.33.0202260959320.29337-100000@se-46.wpa.wtb.tue.nl>; from a.t.hofkamp@tue.nl on Tue, Feb 26, 2002 at 11:58:59AM +0100
References: <20020225025135.GA997@gerg.ca> <Pine.LNX.4.33.0202260959320.29337-100000@se-46.wpa.wtb.tue.nl>
Message-ID: <20020226133714.A26294@noris.de>

Hi,

A.T. Hofkamp:
> I think you are doing a terrific job of answering all issues raised in this
> SIG, many thanks for that.
> 
So do you, right now. Thank you.

> An option object (i.e. an instance of an option class) represents a single
> command-line concept.

I like that concept. It translates rather nicely to interdependent
options -- "build a subclass which handles all of them".

>   The current ripoff program implements the policy that 'the last setting is
>   valid', i.e. if a user specifies 'ripoff --file --pipe', it is considered to
>   be equivalent to 'ripoff --pipe'.
>   I disagree with this equivalence notion. If a user specified both options, it
>   is likely that he wants both options.

I agree.  IMHO it is important not to have silent override options
_in_the_same_context_.

The same story happens with --quiet/--verbose. If somebody specifies "-q
-vv" then that should be an error. However, if the configuration file says
"-v" and the command line "-q", that's OK.

Configuration files, however, are NOT command line options. A config file
should not be forced to look like a command line just so that the command
line parser is happy.

>   Very rough, untested implementation:
> 
Looks sensible, except for the typo:  ;-)
>         return _NOMATH


> - Is the idea of 'an option object represents a single concept' good ?
> - Should we have a seperation between usage and value of an option ?
> 
Yes, and (probably, but that's just me) yes.

> PS With your comparison page, you suggest some form of competition. I do
> not have that intention.

Line count is a (albeit imperfect) measure of complexity.
Command line argument handling should be as simple as possible, but no
simpler. ;-)

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/
-- 
Backup not found: (A)bort (R)etry (P)anic


From a.t.hofkamp@tue.nl  Tue Feb 26 13:26:33 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Tue, 26 Feb 2002 14:26:33 +0100 (CET)
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <20020226133714.A26294@noris.de>
Message-ID: <Pine.LNX.4.33.0202261405020.31857-100000@se-46.wpa.wtb.tue.nl>

On Tue, 26 Feb 2002, Matthias Urlichs wrote:

> The same story happens with --quiet/--verbose. If somebody specifies "-q
> -vv" then that should be an error. However, if the configuration file says
> "-v" and the command line "-q", that's OK.

I agree. The program may 'find' option settings from various places, and
combine them in a well-defined and (preferably) sensible way.
Usually, there is a precedence relation between option settings from different
sources, which is fine as far as I can see.

This does raise the question of other places where options can be found.
- Should the standard package be able to deal with them ?
- Should we use the same syntax, or would a different syntax be allowed ?
- Is a precedence relation ok ?
- How do the various toolkits handle option settings from different sources ?
  (argtools does not currently, at least not in a nice way).

Plenty of questions I think.... :-)

> Configuration files, however, are NOT command line options. A config file
> should not be forced to look like a command line just so that the command
> line parser is happy.

We are on sliding ground here I think. We are very much moving towards a
full-featured command line package that understands any syntax we come up with.

<broad smile>
I knew we would end up with a scanner and parser for our option processing.
</broad smile>


>
> >   Very rough, untested implementation:
> >
> Looks sensible, except for the typo:  ;-)
> >         return _NOMATH

Just a test whether people are reading the code :-)


Tnx for the comment,
Albert
-- 
Constructing a computer program is like writing a painting



From tony@lsl.co.uk  Tue Feb 26 14:20:40 2002
From: tony@lsl.co.uk (Tony J Ibbs (Tibs))
Date: Tue, 26 Feb 2002 14:20:40 -0000
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <Pine.LNX.4.33.0202261405020.31857-100000@se-46.wpa.wtb.tue.nl>
Message-ID: <014d01c1bed0$c4957a10$545aa8c0@lslp862.int.lsl.co.uk>

Hmm. Mattias Urlichs and (I think?) A.T. Hofkamp seem to think that
"contradicting" oneself on the command line - for example::

    $ command -verify -noverify

is a Bad Thing to do. In theory, I'm not sure that's wrong. However, in
any situation where an "alias" has been defined for a command. this sort
of thing can happen quite naturally. It's certainly been not uncommon on
Unix, and I'm sure I remember similar things on VMS as well. Thus, for
example, if the alias is::

    cmd = command -verify

then the user is quite likely to innocently try::

    cmd -noverify

and be surprised when it doesn't work.

Whilst such use of an alias might be deemed bad style, it's difficult to
see how it is entirely preventable, given it's a property of the OS or
shell, not the command itself...

Also, if options are to be taken from several places (configuration
file, command line, etc.), what is the advantage to Optick or whatever
in *knowing* this? Surely the aim should be to pass down arguments in
order (e.g., those from the configuration file first, then from the
command line) without the "command line" module needing to know that any
of the arguments are "special" in any way. And if *that* is the case,
then allowing contradictory options on the command line is exactly
cognate to allowing them between the configuration file and command
line.

I guess I'm saying that I believe we should clearly separate the "get
options" part of the problem (which looks in the various places options
might be and retrieves interesting stuff) from the "use options" part
(which is given the information therefrom and does something with it). I
would then see having two modes of input to the latter - the first
(configuration file oriented) says "here is an individual option and its
value", and the second (for ease of command line handling) says "here is
a command line [1]_, please use it".

.. [1] I'm agnostic over whether "command line" means
   "sequence of 'words'" or "string to be broken up at
   appropriate delimitors".

Hoping this makes sense and isn't just a diversion, Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Give a pedant an inch and they'll take 25.4mm
(once they've established you're talking a post-1959 inch, of course)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)






From a.t.hofkamp@tue.nl  Tue Feb 26 16:20:51 2002
From: a.t.hofkamp@tue.nl (A.T. Hofkamp)
Date: Tue, 26 Feb 2002 17:20:51 +0100 (CET)
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <014d01c1bed0$c4957a10$545aa8c0@lslp862.int.lsl.co.uk>
Message-ID: <Pine.LNX.4.33.0202261650540.32172-100000@se-46.wpa.wtb.tue.nl>

On Tue, 26 Feb 2002, Tony J Ibbs (Tibs) wrote:

> Hmm. Mattias Urlichs and (I think?) A.T. Hofkamp seem to think that
> "contradicting" oneself on the command line - for example::

Yes I do.

>     $ command -verify -noverify
>
> is a Bad Thing to do. In theory, I'm not sure that's wrong. However, in
> any situation where an "alias" has been defined for a command. this sort
> of thing can happen quite naturally. It's certainly been not uncommon on

I didn't realize this, but indeed, it may be the case.

> Whilst such use of an alias might be deemed bad style, it's difficult to
> see how it is entirely preventable, given it's a property of the OS or
> shell, not the command itself...

It is not (and imho also should not).

Your example is very obviously non-compatible with each other. Also, nothing
breaks if the program ignores the 'wrong' option (w.r.t. the user expectations,
that is).
A larger portion of these mutual exclusive options are not so easy to spot, or
are easy to miss in the documentation.
That is why I believe a program should refuse contradictionary options.

On the other hand, that is just me. Others (you included) do not agree with me.


So (and that's the real point afaik), the option processing toolkit should

1) _NOT_ make the choice (this holds at least both for Optik and argtools),
2) the toolkit should make it difficult to let ambiguities like this slip by
   without the programmer realizing (which does argtools better, because of the
   option-object = option-concept approach).

> Also, if options are to be taken from several places (configuration
> file, command line, etc.), what is the advantage to Optick or whatever
> in *knowing* this? Surely the aim should be to pass down arguments in

I am not entirely sure whether an option processing package should be capable
of handling options from various sources, but assuming that is indeed the case:

I consider

$ cmd --verify --noverify

different from

(configfile: --verify)

$ cmd --noverify

In the first case, the user specified (knowingly or not) an ambigious command
line. In the second case, the configfile states that 'verify' should be used,
but the user overrides that on the commandline, i.e. there is no ambiguity.

The implicit assumption in the second case is that options coming from the
configfile have a lower priority than options from the command line.
If I give that information to the option processing package, then the package
can differentiate between the first and the second case, and act differently
(for example, not generate an error in the second case).

Do note that at least argtools does not properly support the second case.
I think that Optik doesn't either (but look in the documentation if you really
want to know).

> I guess I'm saying that I believe we should clearly separate the "get
> options" part of the problem (which looks in the various places options
> might be and retrieves interesting stuff) from the "use options" part
> (which is given the information therefrom and does something with it). I

I agree. I am not even considering usage other than the ability to pull the
results of option processing from the option objects (in the case of argtools).

> would then see having two modes of input to the latter - the first

You mean 'former', the "get options" part ????!!

> (configuration file oriented) says "here is an individual option and its
> value", and the second (for ease of command line handling) says "here is
> a command line [1]_, please use it".
>
> .. [1] I'm agnostic over whether "command line" means
>    "sequence of 'words'" or "string to be broken up at
>    appropriate delimitors".

In fact, the command line is already broken up in pieces before it is delivered
to the program.


Albert
-- 
Constructing a computer program is like writing a painting



From smurf@noris.de  Tue Feb 26 17:00:01 2002
From: smurf@noris.de (Matthias Urlichs)
Date: Tue, 26 Feb 2002 18:00:01 +0100
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <014d01c1bed0$c4957a10$545aa8c0@lslp862.int.lsl.co.uk>; from tony@lsl.co.uk on Tue, Feb 26, 2002 at 02:20:40PM -0000
References: <Pine.LNX.4.33.0202261405020.31857-100000@se-46.wpa.wtb.tue.nl> <014d01c1bed0$c4957a10$545aa8c0@lslp862.int.lsl.co.uk>
Message-ID: <20020226180001.H26294@noris.de>

Hi,

Tony J Ibbs (Tibs):
> is a Bad Thing to do. In theory, I'm not sure that's wrong. However, in
> any situation where an "alias" has been defined for a command. this sort
> of thing can happen quite naturally. It's certainly been not uncommon on
> Unix, and I'm sure I remember similar things on VMS as well. Thus, for
> example, if the alias is::
> 
I see your point.

# alias rm rm -i
# rm -f foo
(typical Unix root-user shell usage)

However, IMHO it's a Bad Idea to configure programs by aliasing them;
one can cause really stupid errors that way.
Programs should be configured by way of a configuration file or similar.

> Also, if options are to be taken from several places (configuration
> file, command line, etc.), what is the advantage to Optick or whatever
> in *knowing* this? Surely the aim should be to pass down arguments in
> order (e.g., those from the configuration file first, then from the
> command line) without the "command line" module needing to know that any
> of the arguments are "special" in any way.

"In order" != "by priority".  It shouldn't matter whether the program
reads the command-line options or the prefs file first, as long as the
former overrides the latter.

In fact, specifying an alternate prefs file by way of an option would be
inconsistent otherwise, since the order of options shouldn't matter.

> Hoping this makes sense and isn't just a diversion, Tibs
> 
It does, though I would not want my options package to know about reading
configuration files. Configuration values may come from many sources
(environment variables, XML files, option=value files, the Registry and
related atrocities), ...). An option processing package might want to
afford some way to feed values into it which may have either lower or
higher priority than the options themselves -- I don't yet have an opinion
whether that really belongs into the option handler -- but it should not
afford One True Way To Store Options Somewhere.

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/
-- 
The price of success in philosophy is triviality.
		-- C. Glymour.


From gward@python.net  Thu Feb 28 00:53:22 2002
From: gward@python.net (Greg Ward)
Date: Wed, 27 Feb 2002 19:53:22 -0500
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <Pine.LNX.4.33.0202260959320.29337-100000@se-46.wpa.wtb.tue.nl>
References: <20020225025135.GA997@gerg.ca> <Pine.LNX.4.33.0202260959320.29337-100000@se-46.wpa.wtb.tue.nl>
Message-ID: <20020228005322.GA11410@gerg.ca>

On 26 February 2002, A.T. Hofkamp said:
> In the argtools-version of the source, you raise a number of questions
> of 'how to do this in argtools' ?  The answer to those questions is
> both low tech (i.e. 'this is how'), and conceptual (i.e. 'this is why
> it is not easily possible').
> 
> Unfortunately, I have little time this week for the first part, so I will
> concentrate on the second part (which is also more interesting here imho).

Yes, the concepts are definitely more important than the mechanics.
Feel free to supply a reworked version of ripoff_argtools.py when you
get the chance, though.  ;-)

> Argtools is built on the following assumption:
> 
> An option object (i.e. an instance of an option class) represents a single
> command-line concept.
> Example, the 'verbosity of the program' (i.e. what level of output should the
> program produce) is a single concept, which should be implemented by a single
> option object.

Got it.  Let me paraphrase, and explain Optik's philosophy in the same
terms:

  * Optik 1.x: an Option represents a particular action performed on
    a particular destination variable

  * Argtools: an Option represents everything you might do to a
    particular destination variable

Eg. in Optik, you might have one Option to increment the 'verbose'
variable ("-vvv"), another Option to set it to a specific value
("--verbose=3"), and a third to zero it ("-q" or "--quiet", which are
synonyms wrapped up in the same Option).  Of course, you're free to
arrange your verbosity options any way you please, but that's how I have
done it so far.

With Argtools, every command-line option that could affect the 'verbose'
variable is bundled into a single Option object.

> From this assumption follows that options are independant from each other
> (since they are related to different concepts), and that the order of options
> is not relevant (also because they control different concepts).
> 
> I think the above are sound design principles. It should be non-trivial to
> deviate from them.

It took a day to sink in, but I very definitely see your point.  There
might very well be an Optik 2.0 after all.  ;-)

>   The current ripoff program implements the policy that 'the last
>   setting is valid', i.e. if a user specifies 'ripoff --file --pipe',
>   it is considered to be equivalent to 'ripoff --pipe'.  I disagree
>   with this equivalence notion. If a user specified both options, it
>   is likely that he wants both options.  By ignoring some of his
>   options, you create unexpected behaviour of the program.  You should
>   at least warn the user that you ignore some part of the commandline,
>   I personally would not even execute the command until the user makes
>   an unambigious choice.

I strongly disagree here; "-qvq" should be the same as "-q", and
"--use-files --use-pipes" should be the same as "--use-pipes".  I do
*not* see those options as mutually exclusive, I see them as writing to
the same variable.  Perhaps "interfering options" would be a better term
for them; Optik is perfectly happy to let you have interfering options,
but the concept is nonsensical with Argtools because its concept of
"option" is somewhat larger.

I'm not sure I can entirely explain my rationale, especially since Tony
Ibbs already hit the main points.  It's based on gut instinct and years
of experience more than anything else.  It boils down to this:
command-line options are processed in left-to-right order, and rightmost
wins.  Order only matters for interfering options, but interfering
options are (IMHO) perfectly OK, because the user typing the command is
not necessarily solely in charge of all the words in that command-line:
witness aliases, or shell scripts that end like

   exec something_else --this --that $*

(Yes, I know that $* is the wrong thing to use in a shell script, but I
can't remember the right spelling.  That's why I don't write shell
scripts.)

>   I think it is better to throw the --file and --pipe options out, and
>   define --output=file and --output=pipe. This is implemented in the
>   EnumOption of argtools. No ambiguities, no user confusion.

I'm lazy.  Why should I have to type that many characters when I can get
away with fewer?  Having 20 command-line options instead of 17 isn't
going to break the bank, and "--file" is just plain easier to type than
"--output=file".  (And "-f", which is what I really use, is easier
still.)

> - Your line count is not entirely fair due to some interfacing problems.
>   If you would develop ripoff using argtools, then you would _NOT_
>   make copies from the parser object to the options object, you would
>   use the parser object itself for querying options.  Alternatively,
>   you would add the option objects into 'options' rather than in the
>   parser object.  In both cases, the lines of code for copying option
>   values should be excluded from the line-count.

But my preferred mode of operation is to pass around an options object
that I can use as follows:

    if options.use_files:
        # ... do things one way ...
    else:
        # ... do them another way

or

    if not options.rip_only:
        # ... encode the track we just ripped ...

I find that the clearest and most obvious way to work, and I'd like to
see more programs work that way.  I most explicitly do *not* want do
have to do this:

    if parser.use_files.Seen():
        # ...

because that takes an implementation detail from one part of the program
-- how exactly the value of the 'use_files' flag is set -- and imposes
it all over the rest of the program.  What if I take your advice and
switch to "--output={file,pipe}" -- do I then have to go throughout my
code and change the above to

    if parser.output.Value() == "file":
        # ...

?  Yuck.  Also, having to call a method just to get an option value is
annoying and clunky.  Direct attribute access is simple, cheap,
painless, and hard to screw up.  (If you get it wrong, Python calls you
on it with AttributeError.)

> - For options with arguments, how to see whether the option is used, etc ?
>   argtools has 3 methods for querying an option object with a value:
>   - Used(): returns 1 if the option is specified, 0 otherwise
>   - Set(): returns 1 if the option on the command line has an argument
>   - Value(): the value of the argument of the option object.
> 
>   Thus:
>   if not option.Used():
>      # option is not specified
>   elif option.Used() and not option.Set():
>      # option without argument is specified, for example --log
>   elif option.Used() and option.Set():
>      # option with argument is specified, for example --log=$HOME/mylog
>      # value of the argument in option.Value()
> 
>   Obviously, if --log is not allowed, then the second case never happens, and
>   Set() and Used() are always the same.
> 
>   Notice the strict seperation of usage and value. In many other option
>   toolkits including Optik, these notions are mixed together into special
>   values of the argument.

Thank you, that clarifies something important.  The lack of "was option
X seen?" bothered me for a while, so I implemented subclasses of
OptionParser and Option to see how hard it was.  (See
examples/required_2.py in the Optik CVS.)  It was trivial.  But I
couldn't see why this was really necessary, because usually all I do is
test a value against None to see if its corresponding option was
supplied.

But you're absolutely right: this is not the right way to see if an
option was supplied; it's only the right way to see if an option value
is None.  Explicit is better than implicit.  Just added it to my to-do
list.

> So the questions are:
> 
> - Is the idea of 'an option object represents a single concept' good ?

Sounds promising, but it will require a careful rethink.  If it happens,
it'll be called Optik 2.0.

> - Should we have a seperation between usage and value of an option ?

Yep, and trivial to add.  Will be in Optik 1.3.

> PS With your comparison page, you suggest some form of competition. I do not
>    have that intention.

I was careful to use the word "comparison" instead of "competition", and
I tried to stay as even-handed as I could.  I'm certainly not aiming for
a competition, I'm trying to get all the good ideas out on the table.
Seems to be working so far.

        Greg
-- 
Greg Ward - geek-at-large                               gward@python.net
http://starship.python.net/~gward/
War is Peace; Freedom is Slavery; Ignorance is Knowledge


From smurf@noris.de  Thu Feb 28 09:05:04 2002
From: smurf@noris.de (Matthias Urlichs)
Date: Thu, 28 Feb 2002 10:05:04 +0100
Subject: [getopt-sig] Comparing option libraries
In-Reply-To: <20020228005322.GA11410@gerg.ca>; from gward@python.net on Wed, Feb 27, 2002 at 07:53:22PM -0500
References: <20020225025135.GA997@gerg.ca> <Pine.LNX.4.33.0202260959320.29337-100000@se-46.wpa.wtb.tue.nl> <20020228005322.GA11410@gerg.ca>
Message-ID: <20020228100504.B26294@noris.de>

Hi,

Greg Ward:
> Eg. in Optik, you might have one Option to increment the 'verbose'
> variable ("-vvv"), another Option to set it to a specific value
> ("--verbose=3")

That's a nice case for optional option arguments. ;-)
So -v3 or --verbose=3 sets verbosity to 3, but a single -v or --verbose
increments it.

I don't think that remembering two options for effectively doing the same
thing would be in any way better.

>    exec something_else --this --that $*
> 
Should be
	exec something_else --this --that "$@"

Yes, the quotes are part of the command line and NOT a mistake.

This is not a problem with shell scripts -- the real problems start when
you try to _parse_ the arguments in the shell.

> [ Calling implementation-detail methods to find out the
>   value of an option ]
> ?  Yuck.

I think I agree here.

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/
-- 
A child is a person who can't understand why someone would give away a
perfectly good kitten.
		-- Doug Larson