[Python-checkins] cpython (3.2): #16152: fix tokenize to ignore whitespace at the end of the code when no
ezio.melotti
python-checkins at python.org
Sat Nov 3 16:51:41 CET 2012
http://hg.python.org/cpython/rev/3ffff1798ed5
changeset: 80191:3ffff1798ed5
branch: 3.2
parent: 80176:cf62fbf2171a
user: Ezio Melotti <ezio.melotti at gmail.com>
date: Sat Nov 03 17:38:43 2012 +0200
summary:
#16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder.
files:
Lib/test/test_tokenize.py | 5 +++++
Lib/tokenize.py | 4 +++-
Misc/ACKS | 1 +
Misc/NEWS | 3 +++
4 files changed, 12 insertions(+), 1 deletions(-)
diff --git a/Lib/test/test_tokenize.py b/Lib/test/test_tokenize.py
--- a/Lib/test/test_tokenize.py
+++ b/Lib/test/test_tokenize.py
@@ -552,6 +552,11 @@
DEDENT '' (4, 0) (4, 0)
DEDENT '' (4, 0) (4, 0)
+Pathological whitespace (http://bugs.python.org/issue16152)
+ >>> dump_tokens("@ ")
+ ENCODING 'utf-8' (0, 0) (0, 0)
+ OP '@' (1, 0) (1, 1)
+
Non-ascii identifiers
>>> dump_tokens("Örter = 'places'\\ngrün = 'green'")
diff --git a/Lib/tokenize.py b/Lib/tokenize.py
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -108,7 +108,7 @@
group("'", r'\\\r?\n'),
r'[bB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
group('"', r'\\\r?\n'))
-PseudoExtras = group(r'\\\r?\n', Comment, Triple)
+PseudoExtras = group(r'\\\r?\n|\Z', Comment, Triple)
PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
def _compile(expr):
@@ -473,6 +473,8 @@
if pseudomatch: # scan for tokens
start, end = pseudomatch.span(1)
spos, epos, pos = (lnum, start), (lnum, end), end
+ if start == end:
+ continue
token, initial = line[start:end], line[start]
if (initial in numchars or # ordinary number
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -71,6 +71,7 @@
Ulf Bartelt
Don Bashford
Nick Bastin
+Ned Batchelder
Jeff Bauer
Mike Bayer
Michael R Bax
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -143,6 +143,9 @@
Library
-------
+- Issue #16152: fix tokenize to ignore whitespace at the end of the code when
+ no newline is found. Patch by Ned Batchelder.
+
- Issue #1207589: Add Cut/Copy/Paste items to IDLE right click Context Menu
Patch by Todd Rovito.
--
Repository URL: http://hg.python.org/cpython
More information about the Python-checkins
mailing list