[Python-checkins] r61979 - in python/trunk: Lib/test/test_tokenize.py Lib/tokenize.py Misc/NEWS
amaury.forgeotdarc
python-checkins at python.org
Fri Mar 28 00:23:54 CET 2008
Author: amaury.forgeotdarc
Date: Fri Mar 28 00:23:54 2008
New Revision: 61979
Modified:
python/trunk/Lib/test/test_tokenize.py
python/trunk/Lib/tokenize.py
python/trunk/Misc/NEWS
Log:
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
"" "" => """", which is invalid code.
Will backport
Modified: python/trunk/Lib/test/test_tokenize.py
==============================================================================
--- python/trunk/Lib/test/test_tokenize.py (original)
+++ python/trunk/Lib/test/test_tokenize.py Fri Mar 28 00:23:54 2008
@@ -487,13 +487,18 @@
>>> roundtrip("# Comment \\\\nx = 0")
True
+Two string literals on the same line
+
+ >>> roundtrip("'' ''")
+ True
+
+Test roundtrip on random python modules.
+pass the '-ucompiler' option to process the full directory.
+
>>>
>>> tempdir = os.path.dirname(f) or os.curdir
>>> testfiles = glob.glob(os.path.join(tempdir, "test*.py"))
- XXX: tokenize doesn not support __future__.unicode_literals yet
- >>> blacklist = ("test_future4.py",)
- >>> testfiles = [f for f in testfiles if not f.endswith(blacklist)]
>>> if not test_support.is_resource_enabled("compiler"):
... testfiles = random.sample(testfiles, 10)
...
Modified: python/trunk/Lib/tokenize.py
==============================================================================
--- python/trunk/Lib/tokenize.py (original)
+++ python/trunk/Lib/tokenize.py Fri Mar 28 00:23:54 2008
@@ -210,12 +210,21 @@
tokval += ' '
if toknum in (NEWLINE, NL):
startline = True
+ prevstring = False
for tok in iterable:
toknum, tokval = tok[:2]
if toknum in (NAME, NUMBER):
tokval += ' '
+ # Insert a space between two consecutive strings
+ if toknum == STRING:
+ if prevstring:
+ tokval = ' ' + tokval
+ prevstring = True
+ else:
+ prevstring = False
+
if toknum == INDENT:
indents.append(tokval)
continue
@@ -244,7 +253,7 @@
t1 = [tok[:2] for tok in generate_tokens(f.readline)]
newcode = untokenize(t1)
readline = iter(newcode.splitlines(1)).next
- t2 = [tok[:2] for tokin generate_tokens(readline)]
+ t2 = [tok[:2] for tok in generate_tokens(readline)]
assert t1 == t2
"""
ut = Untokenizer()
Modified: python/trunk/Misc/NEWS
==============================================================================
--- python/trunk/Misc/NEWS (original)
+++ python/trunk/Misc/NEWS Fri Mar 28 00:23:54 2008
@@ -76,6 +76,10 @@
Library
-------
+- Issue #2495: tokenize.untokenize now inserts a space between two consecutive
+ string literals; previously, ["" ""] was rendered as [""""], which is
+ incorrect python code.
+
- Issue #2248: return the result of the QUIT command. from SMTP.quit().
- Backport of Python 3.0's io module.
More information about the Python-checkins
mailing list