[New-bugs-announce] [issue24194] tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue

[Joshua Landau]

New submission from Joshua Landau:

This is valid:

    ℘· = 1
    print(℘·)
    #>>> 1

But this gives an error token:

    from io import BytesIO
    from tokenize import tokenize

    stream = BytesIO("℘·".encode("utf-8"))
    print(*tokenize(stream.read), sep="\n")
    #>>> TokenInfo(type=56 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
    #>>> TokenInfo(type=53 (ERRORTOKEN), string='℘', start=(1, 0), end=(1, 1), line='℘·')
    #>>> TokenInfo(type=53 (ERRORTOKEN), string='·', start=(1, 1), end=(1, 2), line='℘·')
    #>>> TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')


This is a continuation of http://bugs.python.org/issue9712. I'm not able to reopen the issue, so I thought I should report it anew.

It is tokenize that is wrong - Other_ID_Start and Other_ID_Continue are documented to be valid:

https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers

----------
components: Library (Lib)
messages: 243188
nosy: Joshua.Landau
priority: normal
severity: normal
status: open
title: tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue
type: behavior
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue24194>
_______________________________________