[issue25324] Importing tokenize modifies token

[Albert-Jan Nijburg]

Albert-Jan Nijburg added the comment:

I've updated the PR and added the tokenize tokens to token.h and their names to tokenizer.c. This way they'll show up when you run token.py. The names will always be in tok_name and tokenizer.py will use those. Not breaking the public api and no longer modifying token.py when you import tokenizer.py.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25324>
_______________________________________