[docs] [issue2134] Add new attribute to TokenInfo to report specific token IDs

[Meador Inge]

Meador Inge <meadori at gmail.com> added the comment:

The proposed documentation text seems too complicated and language expert speaky to me.  We should try to link to standard definitions when possible to reduce the text here.  For example, I believe the "Operators" and "Delimiters" tokens in the "Lexical Analysis" section of the docs (http://docs.python.org/dev/reference/lexical_analysis.html#operators) are exactly what we are trying to describe when referencing "literal tokens" and "affected tokens".

I like Nick's idea to introduce a new attribute for the exact type, while keeping the tuple structure itself backwards compatible.  Attached is a patch for 3.3 that updates the docs, adds exact_type, adds new unit tests, and adds a new CLI option for displaying token names using the exact type.

An example of the new CLI option is:

$ echo '1+2**4' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'            
1,1-1,2:            OP             '+'            
1,2-1,3:            NUMBER         '2'            
1,3-1,5:            OP             '**'           
1,5-1,6:            NUMBER         '4'            
1,6-1,7:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''             
$ echo '1+2**4' | ./python -m tokenize -e
1,0-1,1:            NUMBER         '1'            
1,1-1,2:            PLUS           '+'            
1,2-1,3:            NUMBER         '2'            
1,3-1,5:            DOUBLESTAR     '**'           
1,5-1,6:            NUMBER         '4'            
1,6-1,7:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''

----------
Added file: http://bugs.python.org/file24045/tokenize-exact-type-v0.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue2134>
_______________________________________