[New-bugs-announce] [issue13832] tokenization assuming ASCII whitespace; missing multiline case

[Jim Jewett]

New submission from Jim Jewett <jimjjewett at gmail.com>:

Parser/parsetok.c was recently changed (e.g. http://hg.python.org/cpython/rev/2bd7f40108b4 ) to raise an error if multiple statements were found in a single-statement compile call.  It sensibly ignores trailing whitespace and comments.  Unfortunately,

(1)  It looks only at (c == ' ' || c == '\t' || c == '\n' || c == '\014') as opposed to using Py_UNICODE_ISSPACE(ch)
(2)  It assumes that a "#" means the rest of the line is OK, instead of looking for additional linebreaks.

Not sure whether to mark this a bug or an enhancement, since it is already strictly better than the 3.2 behavior of never warning about extra text.

----------
components: Interpreter Core
messages: 151652
nosy: Jim.Jewett
priority: normal
severity: normal
status: open
title: tokenization assuming ASCII whitespace; missing multiline case
versions: Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13832>
_______________________________________