[issue25324] Importing tokenize modifies token

[STINNER Victor]

STINNER Victor added the comment:

> I would fix this by making tokenize.tok_name a copy. It looks like this behaviour dates back to 1997 (see revision 1efc4273fdb7).

token.tok_name is part of the Python public API:
https://docs.python.org/dev/library/token.html#token.tok_name

whereas tokenize.tok_name isn't documented. So I dislike having two disconnected mappings. I prefer to add tokenize tokens directly in Lib/token.py, and then get COMMENT, NL and ENCODING using tok_name.index().

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25324>
_______________________________________