[Python-checkins] r75430 - in python/trunk/Lib: test/test_tokenize.py tokenize.py

[benjamin.peterson]

Author: benjamin.peterson
Date: Thu Oct 15 03:49:37 2009
New Revision: 75430

Log:
use floor division and add a test that exercises the tabsize codepath

Modified:
   python/trunk/Lib/test/test_tokenize.py
   python/trunk/Lib/tokenize.py

Modified: python/trunk/Lib/test/test_tokenize.py
==============================================================================
--- python/trunk/Lib/test/test_tokenize.py	(original)
+++ python/trunk/Lib/test/test_tokenize.py	Thu Oct 15 03:49:37 2009
@@ -508,6 +508,23 @@
     ...         break
     ... else: True
     True
+
+Evil tabs
+    >>> dump_tokens("def f():\\n\\tif x\\n        \tpass")
+    NAME       'def'         (1, 0) (1, 3)
+    NAME       'f'           (1, 4) (1, 5)
+    OP         '('           (1, 5) (1, 6)
+    OP         ')'           (1, 6) (1, 7)
+    OP         ':'           (1, 7) (1, 8)
+    NEWLINE    '\\n'          (1, 8) (1, 9)
+    INDENT     '\\t'          (2, 0) (2, 1)
+    NAME       'if'          (2, 1) (2, 3)
+    NAME       'x'           (2, 4) (2, 5)
+    NEWLINE    '\\n'          (2, 5) (2, 6)
+    INDENT     '         '   (3, 0) (3, 9)
+    NAME       'pass'        (3, 9) (3, 13)
+    DEDENT     ''            (4, 0) (4, 0)
+    DEDENT     ''            (4, 0) (4, 0)
 """
 
 

Modified: python/trunk/Lib/tokenize.py
==============================================================================
--- python/trunk/Lib/tokenize.py	(original)
+++ python/trunk/Lib/tokenize.py	Thu Oct 15 03:49:37 2009
@@ -319,7 +319,7 @@
                 if line[pos] == ' ':
                     column = column + 1
                 elif line[pos] == '\t':
-                    column = (column/tabsize + 1)*tabsize
+                    column = (column//tabsize + 1)*tabsize
                 elif line[pos] == '\f':
                     column = 0
                 else: