[Python-checkins] cpython (2.7): Issue #20387: Backport fix from Python 3.4
jason.coombs
python-checkins at python.org
Sun Jun 28 19:08:58 CEST 2015
https://hg.python.org/cpython/rev/cb9df1ae287b
changeset: 96716:cb9df1ae287b
branch: 2.7
user: Jason R. Coombs <jaraco at jaraco.com>
date: Sun Jun 28 13:05:19 2015 -0400
summary:
Issue #20387: Backport fix from Python 3.4
files:
Lib/tokenize.py | 17 +++++++++++++++++
Misc/NEWS | 3 +++
2 files changed, 20 insertions(+), 0 deletions(-)
diff --git a/Lib/tokenize.py b/Lib/tokenize.py
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -198,6 +198,8 @@
def untokenize(self, iterable):
it = iter(iterable)
+ indents = []
+ startline = False
for t in it:
if len(t) == 2:
self.compat(t, it)
@@ -205,6 +207,21 @@
tok_type, token, start, end, line = t
if tok_type == ENDMARKER:
break
+ if tok_type == INDENT:
+ indents.append(token)
+ continue
+ elif tok_type == DEDENT:
+ indents.pop()
+ self.prev_row, self.prev_col = end
+ continue
+ elif tok_type in (NEWLINE, NL):
+ startline = True
+ elif startline and indents:
+ indent = indents[-1]
+ if start[1] >= len(indent):
+ self.tokens.append(indent)
+ self.prev_col = len(indent)
+ startline = False
self.add_whitespace(start)
self.tokens.append(token)
self.prev_row, self.prev_col = end
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -30,6 +30,9 @@
Library
-------
+- Issue #20387: Restore semantic round-trip correctness in tokenize/untokenize
+ for tab-indented blocks.
+
- Issue #24456: Fixed possible buffer over-read in adpcm2lin() and lin2adpcm()
functions of the audioop module. Fixed SystemError when the state is not a
tuple. Fixed possible memory leak.
--
Repository URL: https://hg.python.org/cpython
More information about the Python-checkins
mailing list