[issue25643] Python tokenizer rewriting

[STINNER Victor]

STINNER Victor added the comment:

"especially for issues with null byte"

I don't think that we should put to much energy in handling correctly NUL bytes. I see NUL bytes in code as bugs in the code, not in the Python parser. We *might* try to give warnings or better error messages to the user, that's all.

----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25643>
_______________________________________