[issue25324] Importing tokenize modifies token

[STINNER Victor]

STINNER Victor added the comment:

Another example from Tools/i18n/pygettext.py, TokenEater::

    def __call__(self, ttype, tstring, stup, etup, line):
        # dispatch
##        import token
##        print >> sys.stderr, 'ttype:', token.tok_name[ttype], \
##              'tstring:', tstring
        self.__state(ttype, tstring, stup[0])


...

            eater.set_filename(filename)
            try:
                tokens = tokenize.tokenize(fp.readline)
                for _token in tokens:
                    eater(*_token)

Another example using token.tok_name with token types coming from tokenize.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25324>
_______________________________________