[issue12675] tokenize module happily tokenizes code with syntax errors

[Terry J. Reedy]

Terry J. Reedy <tjreedy at udel.edu> added the comment:

I have not used tokenize, but if it is *not* intended to exactly reproduce the internal tokenizer  behavior, the claim that it is should be amended.

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12675>
_______________________________________