[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

[Ned Batchelder]

Ned Batchelder added the comment:

Updated with new (better) patch, for v2.7 and v3.3.  They are the same except for the test.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16152>
_______________________________________