A simple single line, triple-quoted comment is giving syntax error. Why?

[Tim Chase]

On 2015-03-27 10:15, Steven D'Aprano wrote:
> If that's all it is, why don't you just run the tokenizer over it
> and see what it says?
> 
> py> from cStringIO import StringIO
> py> code = StringIO('spam = "abcd" "efgh"\n')
> py> import tokenize
> py> for item in tokenize.generate_tokens(code.readline):
> ...     print item
> ...
> (1, 'spam', (1, 0), (1, 4), 'spam = "abcd" "efgh"\n')
> (51, '=', (1, 5), (1, 6), 'spam = "abcd" "efgh"\n')
> (3, '"abcd"', (1, 7), (1, 13), 'spam = "abcd" "efgh"\n')
> (3, '"efgh"', (1, 14), (1, 20), 'spam = "abcd" "efgh"\n')
> (4, '\n', (1, 20), (1, 21), 'spam = "abcd" "efgh"\n')
> (0, '', (2, 0), (2, 0), '')
> 
> 
> Looks to me that the two string literals each get their own token,

Nice.  I haven't played with the tokenize module before, but
resolving arguments on comp.lang.python is one of the best possible
uses.

It was interesting to try other feeders to generate_tokens(), my favorite being

>>> import tokenize
>>> i = iter(["spam = 'abc' 'def'"])
>>> for item in tokenize.generate_tokens(lambda: next(i)):
...     print(item)
... 
TokenInfo(type=1 (NAME), string='spam', start=(1, 0), end=(1, 4), line="spam = 'abc' 'def'")
TokenInfo(type=52 (OP), string='=', start=(1, 5), end=(1, 6), line="spam = 'abc' 'def'")
TokenInfo(type=3 (STRING), string="'abc'", start=(1, 7), end=(1, 12), line="spam = 'abc' 'def'")
TokenInfo(type=3 (STRING), string="'def'", start=(1, 13), end=(1, 18), line="spam = 'abc' 'def'")
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')


It's also nice to have the translation from token-type to token-type-name in Py3

-tkc