[issue2134] Add new attribute to TokenInfo to report specific token IDs

[Nick Coghlan]

Nick Coghlan <ncoghlan at gmail.com> added the comment:

Sure, but what does that have to do with anything? tokenize isn't a general purpose tokenizer, it's specifically for tokenizing Python source code.

The *problem* is that it doesn't currently fully tokenize everything, but doesn't explicitly say that in the module documentation.

Hence my proposed two-fold fix: document the current behaviour explicitly and also add a separate "exact_type" attribute for easy access to the detailed tokenization without doing your own string comparisons.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2134>
_______________________________________