A simple single line, triple-quoted comment is giving syntax error. Why?

[Steven D'Aprano]

On Fri, 27 Mar 2015 05:56 am, Thomas 'PointedEars' Lahn wrote:
[snip argument]


Hey guys, I'm trying to follow the argument but I must admit you've
completely lost me with the angels-on-the-head-of-a-pin nitpicking about
EBNF. I love a good pedant-brawl as much as you, so let me see if I've got
this straight, correct me if I'm wrong.

You're arguing whether or not in the following line of code:

spam = "abcd" "efgh"
# implicitly concatenated to "abcdefgh" at compile time

the right hand side pair of strings counts as a single token or two? Am I
right, or am I missing something?

If that's all it is, why don't you just run the tokenizer over it and see
what it says?

py> from cStringIO import StringIO
py> code = StringIO('spam = "abcd" "efgh"\n')
py> import tokenize
py> for item in tokenize.generate_tokens(code.readline):
...     print item
...
(1, 'spam', (1, 0), (1, 4), 'spam = "abcd" "efgh"\n')
(51, '=', (1, 5), (1, 6), 'spam = "abcd" "efgh"\n')
(3, '"abcd"', (1, 7), (1, 13), 'spam = "abcd" "efgh"\n')
(3, '"efgh"', (1, 14), (1, 20), 'spam = "abcd" "efgh"\n')
(4, '\n', (1, 20), (1, 21), 'spam = "abcd" "efgh"\n')
(0, '', (2, 0), (2, 0), '')


Looks to me that the two string literals each get their own token, and are
concatenated at a later stage of compilation, not during parsing.



-- 
Steven