[issue31394] Ellipsis_token.type != token.ELLIPSIS

[Berker Peksag]

Berker Peksag added the comment:

Please don't reopen an issue if it was closed by a core developer.

It's not clear to me what exactly do you want to change in the implementation or documentation.

     A sequence of three periods has a special meaning as an ellipsis literal.

is literally describes how the ELLIPSIS token is identified in Parser/tokenizer.c (see PyToken_ThreeChars in that file) So a sequence of three periods is identified as an ellipsis literal which is an expression in Python.

Do you want to change tokenize.tokenize() (it's in Lib/tokenize.py) so it will return

    TokenInfo(type=52 (ELLIPSIS), ...)

instead of

    TokenInfo(type=53 (OP), ...)

? Note that ELLIPSIS has been added to tokenize.EXACT_TOKEN_TYPES in issue 24622. To me, since it has been added to tokenize.EXACT_TOKEN_TYPES there is no need to special case ELLIPSIS in Lib/tokenize.py.

Or do you want to clarify

    To simplify token stream handling, all Operators and Delimiters tokens are returned using the generic token.OP token type.

at https://docs.python.org/3/library/tokenize.html?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31394>
_______________________________________