[issue31394] Ellipsis_token.type != token.ELLIPSIS

[Aivar Annamaa]

Aivar Annamaa added the comment:

(Sorry, I didn't mean to challenge the authority of a core developer. I simply didn't notice that adding a comment reopens the issue. I hope this time I selected correct parameters and this doesn't happen again)

I'm trying to rephrase my concern. 

Initially I thought there was a mistake in the tokenizer or in the token module.

After you pointed out the documentation about token.OP and exact_type, I'm worried about a smaller detail. The documentation only talks about *operators* and *delimiters* having type attribute set to token.OP. According to my understanding (and also the listings at https://docs.python.org/3/reference/lexical_analysis.html#operators), ellipsis is neither operator nor delimiter. 

Am I right about this? 

(I understand that the source representation of both ELLIPSIS and DOT tokens contains period *character(s)*, but I don't see why is this relevant when we discuss the properties of *tokens*)

Anyway, if ellipsis is neither operator nor delimiter, (and if there is a reason why it is treated similarly with operators and delimiters) then I recommend to update the documentation by replacing 

> all Operators and Delimiters tokens 
> are returned using the generic token.OP token type

with 

> all Operators, Delimiters and Ellipsis tokens
> are returned using the generic token.OP token type


(I understand, that this is not a serious issue. If you prefer not to discuss it further then I'm happy to leave it as it is.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31394>
_______________________________________