shlex.split != shlex.shlex get_token til eof

[p.lavarre at ieee.org]

How can I instantiate shlex.shlex to behave like shlex.split does?

I see shlex.split gives me what I want:

import shlex
print shlex.split("1.2e+3")[0]          # 1.2e+3

But every doc'ed instantiation of shlex.shlex surprisingly gives me
something else:

s1 = shlex.shlex("1.2e+3", None, False)
print s1.get_token()                    # 1
s2 = shlex.shlex("1.2e+3", None, True)
print s2.get_token()                    # 1

I can get closer to the shlex.split state by hacking the shlex.shlex
state before the first get_token invocation:

s3 = shlex.shlex("1.2e+3", None, True)
s3.wordchars += ".+-"
print s3.get_token()                    # 1.2e+3

But how can I know if I've discovered the last hack that I need?

I ask because I'm patching a cmd.Cmd app to tweak the interpretation of
tokens. It was calling shlex.split. My class instead subclasses
shlex.shlex to catch the context of instream.tell().  That context lets
me see the significant differences between such sources as r'0xAB' and
r'0x"AB"'.

Can I somehow override just what I mean to change in shlex.split, and
not more?

Curiously yours, thanks in advance, Pat LaVarre
http://docs.python.org/lib/module-shlex.html
2.4.2#67 Python, 2.3.5#1 Python