Parsing parameters with quotes

[John Machin]

Alex Martelli <aleax at aleax.it> wrote in message news:<l5hca.55547$pG1.1276278 at news1.tin.it>...
]
> 
> and similarly for module tokenize:
> 
> import tokenize
> print [t[1] for t in tokenize.generate_tokens(si(xx).readline) if t[1]]
> 
> emits:
> 
> ['foo', '"this is one"', 'and', 'this', 'is', 'not']

Using a programming-language tokeni[sz]er on a different language can
be as fraught with peril as attempting to infer user requirements from
one example.
As far as the latter is concerned, I would however be highly surprised
if the OP's non-quoted tokens consisted only of Python identifiers
and/or if the handling of spaces in the following example matched what
the OP had in mind.

>>> import tokenize, token
>>> for t in tokenize.generate_tokens(iter(['  x  y  ']).next): print
token.tok_name[t[0]], repr(t[1])
...
INDENT '  '
NAME 'x'
NAME 'y'
ERRORTOKEN ' '
ERRORTOKEN ' '
>>> for t in tokenize.generate_tokens(iter(['  x  y  --help']).next):
print token.tok_name[t[0]], repr(t[1])
...
INDENT '  '
NAME 'x'
NAME 'y'
OP '-'
OP '-'
NAME 'help'