Weird bahaviour from shlex - line no

[Peter Otten]

Dave Angel wrote:

> On 28/9/2013 02:26, Daniel Stojanov wrote:
> 
>> Can somebody explain this. The line number reported by shlex depends
>> on the previous token. I want to be able to tell if I have just popped
>> the last token on a line.
>>
> 
> I agree that it seems weird.  However, I don't think you have made
> clear why it's not what you (and I) expect.
> 
> import shlex
> 
> def parseit(string):
>     print
>     print "Parsing -", string
>     first = shlex.shlex(string)
>     token = "dummy"
>     while token:
>         token = first.get_token()
>         print token, " -- line", first.lineno
> 
> parseit("word1 word2\nword3")     #first
> parseit("word1 word2,\nword3")    #second
> parseit("word1 word2,word3\nword4")
> parseit("word1 word2+,?\nword3")
> 
> This will display the lineno attribute for every token.
> 
> shlex is documented at:
> 
> http://docs.python.org/2/library/shlex.html
> 
> And lineno is documented on that page as:
> 
> """shlex.lineno
> Source line number (count of newlines seen so far plus one).
> """
> 
> It's not at all clear what "seen so far" is intended to mean, but in
> practice, the line number is incremented for the last token on the
> line. Thus your first example
> 
> Parsing - word1 word2
> word3
> word1  -- line 1
> word2  -- line 2
> word3  -- line 2
>   -- line 2
> 
> word2 has the incremented line number.
> 
> But when the token is neither whitespace nor ASCII letters, then it
> doesn't increment lineno.  Thus second example:
> 
> Parsing - word1 word2,
> word3
> word1  -- line 1
> word2  -- line 1
> ,  -- line 1                      #we would expect this to be "line 2"
> word3 -- line 2 -- line 2
> 
> Anybody else have some explanation 

The explanation seems obvious: a word may be continued by the next character 
if that is in wordchars, so the parser has to look at that character. If it 
happens to be '\n' the lineno is immediately incremented. Non-wordchars are 
returned as single characters, so there is no need to peek ahead and the 
lineno is not altered.

In short: this looks like an implementation accident. 

OP: I don't see a usecase for the current behaviour -- I suggest that you 
file a bug report.

> or advice for Daniel, other than
> preprocessing the string by stripping any non letters off the end of the
> line?

The following gives the tokens' starting line for your examples

def shlexiter(s):
    p = shlex.shlex(s)
    p.whitespace = p.whitespace.replace("\n", "")
    while True:
        lineno = p.lineno
        token = p.get_token()
        if not token:
            break
        if token == "\n":
            continue
        yield lineno, token

def parseit(string):
    print("Parsing - {!r}".format(string))
    for lineno, token in shlexiter(string):
        print("{:3} {!r}".format(lineno, token))
    print("")

but I have no idea about the implications for more complex input.