[issue5857] Return namedtuples from tokenize token generator

[Georg Brandl]

Georg Brandl <georg at python.org> added the comment:

Note that in tokenize.diff, "TokenInfo" should be in __all__ instead of
"Token".  I agree with Raymond on the inner tuples.

----------
nosy: +georg.brandl

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5857>
_______________________________________