[issue19035] tokenize.generate_tokens treat '\f' symbol as the end of file (when reading in unicode)

[R. David Murray]

R. David Murray added the comment:

I suspect this isn't the only place where the change in what is considered a (unicode) line ending character between 2.6 and 2.7/python3 is an issue.  As you observe, it causes very subtle bugs.  I'm going to have to go trolling through the python3 email package looking for places where this could break things :(.

----------
nosy: +r.david.murray

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19035>
_______________________________________