[issue24622] tokenize.py: missing EXACT_TOKEN_TYPES

[Aaron Meurer]

Aaron Meurer <asmeurer at gmail.com> added the comment:

I would suggest adding this to the what's new document https://docs.python.org/3.7/whatsnew/3.7.html. The change affects user-facing code (the exact_type attribute of TokenInfo is OP for ... and -> tokens prior to this patch). 

I would also point out that this directly contradicts the documentation (from https://docs.python.org/3/library/tokenize.html, "To simplify token stream handling, all operator and delimiter tokens and Ellipsis are returned using the generic OP token type. The exact type can be determined by checking the exact_type property on the named tuple returned from tokenize.tokenize()."), so I don't see why it can't be backported.

----------
nosy: +Aaron.Meurer

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue24622>
_______________________________________