Weird bahaviour from shlex - line no

[Andreas Perstinger]

On 28.09.2013 08:26, Daniel Stojanov wrote:
> Can somebody explain this. The line number reported by shlex depends
> on the previous token. I want to be able to tell if I have just popped
> the last token on a line.
>
[SNIP]
>
> second = shlex.shlex("word1 word2,\nword3")

Punctuation characters like the comma are not considered as word 
characters by default and thus are seen as different tokens (consisting 
of only a single character):

 >>> lexer = shlex.shlex("foo, bar, ...")
 >>> token = lexer.get_token()
 >>> while token != lexer.eof:
...   print token
...   token = lexer.get_token()
...
foo
,
bar
,
.
.
.

If you want to treat them as "word" characters you need to add them to 
the string "wordchars" (a public attribute of the "shlex" instance):

 >>> lexer = shlex.shlex("foo.bar, baz")
 >>> lexer.wordchar += '.,'
 >>> print lexer.get_token()
foo.bar,
 >>> print lexer.get_token()
baz

There is also a "debug" attribute (with three different levels: 1, 2, 3; 
default value 0 means no debug output):

 >>> lexer = shlex.shlex("foo, bar, ...")
 >>> lexer.debug = 1
 >>> print lexer.get_token()
shlex: token='foo'
foo
 >>> print lexer.get_token()
shlex: popping token ','
,

Bye, Andreas