Fri Oct 15 01:00:01 CEST 2010

Author: benjamin.peterson
Date: Fri Oct 15 01:00:00 2010
New Revision: 85509

Log:
Merged revisions 83852-83853,83857,84042,84216,84274-84276,84375,85388,85478,85506-85508 via svnmerge from 
svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

........
  r83852 | benjamin.peterson | 2010-08-08 15:45:44 -0500 (Sun, 08 Aug 2010) | 1 line
  
  wrap with parens
........
  r83853 | benjamin.peterson | 2010-08-08 15:46:31 -0500 (Sun, 08 Aug 2010) | 1 line
  
  use parens
........
  r83857 | benjamin.peterson | 2010-08-08 15:59:49 -0500 (Sun, 08 Aug 2010) | 1 line
  
  things which use touch_import should be pre order
........
  r84042 | george.boutsioukis | 2010-08-14 16:10:19 -0500 (Sat, 14 Aug 2010) | 2 lines
  
  This revision incorporates into the 2to3 tool the new, faster, tree matching algorithm developed during a GSOC project. The algorithm resides in the two added modules, btm_matcher and btm_utils. New code has been added to drive the new matching process in refactor.py and a few minor changes were made in other modules. A BM_compatible flag(False by default) has been added in fixer_base and it is set to True in most of the current fixers.
........
  r84216 | benjamin.peterson | 2010-08-19 16:44:05 -0500 (Thu, 19 Aug 2010) | 1 line
  
  allow star_expr in testlist_gexp
........
  r84274 | benjamin.peterson | 2010-08-22 18:40:46 -0500 (Sun, 22 Aug 2010) | 1 line
  
  wrap long line
........
  r84275 | benjamin.peterson | 2010-08-22 18:42:22 -0500 (Sun, 22 Aug 2010) | 1 line
  
  cleanup
........
  r84276 | benjamin.peterson | 2010-08-22 18:51:01 -0500 (Sun, 22 Aug 2010) | 1 line
  
  when there's a None value and a traceback, don't call type with it #9661
........
  r84375 | george.boutsioukis | 2010-08-31 08:38:53 -0500 (Tue, 31 Aug 2010) | 3 lines
  
  Idiomatic code changes & stylistic issues fixed in the BottomMatcher module. Thanks to Benjamin Peterson for taking the time to review the code.
........
  r85388 | benjamin.peterson | 2010-10-12 17:27:44 -0500 (Tue, 12 Oct 2010) | 1 line
  
  fix urllib fixer with multiple as imports on a line #10069
........
  r85478 | benjamin.peterson | 2010-10-14 08:09:56 -0500 (Thu, 14 Oct 2010) | 1 line
  
  stop abusing docstrings
........
  r85506 | benjamin.peterson | 2010-10-14 17:45:19 -0500 (Thu, 14 Oct 2010) | 1 line
  
  kill sibling import
........
  r85507 | benjamin.peterson | 2010-10-14 17:54:15 -0500 (Thu, 14 Oct 2010) | 1 line
  
  remove trailing whitespace
........
  r85508 | benjamin.peterson | 2010-10-14 17:55:28 -0500 (Thu, 14 Oct 2010) | 1 line
  
  typo
........


Added:
   python/branches/release27-maint/Lib/lib2to3/btm_matcher.py
      - copied unchanged from r85508, /sandbox/trunk/2to3/lib2to3/btm_matcher.py
   python/branches/release27-maint/Lib/lib2to3/btm_utils.py
      - copied unchanged from r85508, /sandbox/trunk/2to3/lib2to3/btm_utils.py
Modified:
   python/branches/release27-maint/Lib/lib2to3/   (props changed)
   python/branches/release27-maint/Lib/lib2to3/Grammar.txt
   python/branches/release27-maint/Lib/lib2to3/fixer_base.py
   python/branches/release27-maint/Lib/lib2to3/fixer_util.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_apply.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_basestring.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_buffer.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_callable.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_dict.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_except.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_exec.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_execfile.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_exitfunc.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_filter.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_funcattrs.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_future.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_getcwdu.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_has_key.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_idioms.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_import.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_imports.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_input.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_intern.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_isinstance.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools_imports.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_long.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_map.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_metaclass.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_methodattrs.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_next.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_nonzero.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_operator.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_paren.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_print.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_raise.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_raw_input.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_reduce.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_renames.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_repr.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_set_literal.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_standarderror.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_sys_exc.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_throw.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_tuple_params.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_types.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_unicode.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_urllib.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_xrange.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_xreadlines.py
   python/branches/release27-maint/Lib/lib2to3/fixes/fix_zip.py
   python/branches/release27-maint/Lib/lib2to3/patcomp.py
   python/branches/release27-maint/Lib/lib2to3/pygram.py
   python/branches/release27-maint/Lib/lib2to3/pytree.py
   python/branches/release27-maint/Lib/lib2to3/refactor.py
   python/branches/release27-maint/Lib/lib2to3/tests/data/bom.py
   python/branches/release27-maint/Lib/lib2to3/tests/test_fixers.py
   python/branches/release27-maint/Lib/lib2to3/tests/test_pytree.py

Modified: python/branches/release27-maint/Lib/lib2to3/Grammar.txt
==============================================================================

--- python/branches/release27-maint/Lib/lib2to3/Grammar.txt	(original)
+++ python/branches/release27-maint/Lib/lib2to3/Grammar.txt	Fri Oct 15 01:00:00 2010
@@ -128,7 +128,7 @@
        '`' testlist1 '`' |
        NAME | NUMBER | STRING+ | '.' '.' '.')
 listmaker: (test|star_expr) ( comp_for | (',' (test|star_expr))* [','] )
-testlist_gexp: test ( comp_for | (',' (test|star_expr))* [','] )
+testlist_gexp: (test|star_expr) ( comp_for | (',' (test|star_expr))* [','] )
 lambdef: 'lambda' [varargslist] ':' test
 trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
 subscriptlist: subscript (',' subscript)* [',']

Modified: python/branches/release27-maint/Lib/lib2to3/fixer_base.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixer_base.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixer_base.py	Fri Oct 15 01:00:00 2010
@@ -24,6 +24,7 @@
 
     PATTERN = None  # Most subclasses should override with a string literal
     pattern = None  # Compiled pattern, set by compile_pattern()
+    pattern_tree = None # Tree representation of the pattern
     options = None  # Options object passed to initializer
     filename = None # The filename (set by set_filename)
     logger = None   # A logger (set by set_filename)
@@ -36,6 +37,12 @@
     _accept_type = None # [Advanced and not public] This tells RefactoringTool
                         # which node type to accept when there's not a pattern.
 
+    keep_line_order = False # For the bottom matcher: match with the
+                            # original line order
+    BM_compatible = False # Compatibility with the bottom matching
+                          # module; every fixer should set this
+                          # manually
+
     # Shortcut for access to Python grammar symbols
     syms = pygram.python_symbols
 
@@ -58,7 +65,9 @@
         self.{pattern,PATTERN} in .match().
         """
         if self.PATTERN is not None:
-            self.pattern = PatternCompiler().compile_pattern(self.PATTERN)
+            PC = PatternCompiler()
+            self.pattern, self.pattern_tree = PC.compile_pattern(self.PATTERN,
+                                                                 with_tree=True)
 
     def set_filename(self, filename):
         """Set the filename, and a logger derived from it.

Modified: python/branches/release27-maint/Lib/lib2to3/fixer_util.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixer_util.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixer_util.py	Fri Oct 15 01:00:00 2010
@@ -295,8 +295,8 @@
     """ Works like `does_tree_import` but adds an import statement
         if it was not imported. """
     def is_import_stmt(node):
-        return node.type == syms.simple_stmt and node.children and \
-               is_import(node.children[0])
+        return (node.type == syms.simple_stmt and node.children and
+                is_import(node.children[0]))
 
     root = find_root(node)
 
@@ -319,8 +319,8 @@
     # if that also fails, we stick to the beginning of the file
     if insert_pos == 0:
         for idx, node in enumerate(root.children):
-            if node.type == syms.simple_stmt and node.children and \
-               node.children[0].type == token.STRING:
+            if (node.type == syms.simple_stmt and node.children and
+               node.children[0].type == token.STRING):
                 insert_pos = idx + 1
                 break
 

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_apply.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_apply.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_apply.py	Fri Oct 15 01:00:00 2010
@@ -12,6 +12,7 @@
 from ..fixer_util import Call, Comma, parenthesize
 
 class FixApply(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
     power< 'apply'

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_basestring.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_basestring.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_basestring.py	Fri Oct 15 01:00:00 2010
@@ -6,6 +6,7 @@
 from ..fixer_util import Name
 
 class FixBasestring(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = "'basestring'"
 

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_buffer.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_buffer.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_buffer.py	Fri Oct 15 01:00:00 2010
@@ -9,6 +9,7 @@
 
 
 class FixBuffer(fixer_base.BaseFix):
+    BM_compatible = True
 
     explicit = True # The user must ask for this fixer
 

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_callable.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_callable.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_callable.py	Fri Oct 15 01:00:00 2010
@@ -11,6 +11,9 @@
 from lib2to3.fixer_util import Call, Name, String, Attr, touch_import
 
 class FixCallable(fixer_base.BaseFix):
+    BM_compatible = True
+
+    order = "pre"
 
     # Ignore callable(*args) or use of keywords.
     # Either could be a hint that the builtin callable() is not being used.

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_dict.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_dict.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_dict.py	Fri Oct 15 01:00:00 2010
@@ -40,6 +40,8 @@
 
 
 class FixDict(fixer_base.BaseFix):
+    BM_compatible = True
+
     PATTERN = """
     power< head=any+
          trailer< '.' method=('keys'|'items'|'values'|

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_except.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_except.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_except.py	Fri Oct 15 01:00:00 2010
@@ -34,6 +34,7 @@
                 yield (n, nodes[i+2])
 
 class FixExcept(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
     try_stmt< 'try' ':' (simple_stmt | suite)

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_exec.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_exec.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_exec.py	Fri Oct 15 01:00:00 2010
@@ -16,6 +16,7 @@
 
 
 class FixExec(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
     exec_stmt< 'exec' a=any 'in' b=any [',' c=any] >

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_execfile.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_execfile.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_execfile.py	Fri Oct 15 01:00:00 2010
@@ -13,6 +13,7 @@
 
 
 class FixExecfile(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
     power< 'execfile' trailer< '(' arglist< filename=any [',' globals=any [',' locals=any ] ] > ')' > >

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_exitfunc.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_exitfunc.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_exitfunc.py	Fri Oct 15 01:00:00 2010
@@ -9,6 +9,8 @@
 
 
 class FixExitfunc(fixer_base.BaseFix):
+    keep_line_order = True
+    BM_compatible = True
 
     PATTERN = """
               (

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_filter.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_filter.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_filter.py	Fri Oct 15 01:00:00 2010
@@ -19,6 +19,7 @@
 from ..fixer_util import Name, Call, ListComp, in_special_context
 
 class FixFilter(fixer_base.ConditionalFix):
+    BM_compatible = True
 
     PATTERN = """
     filter_lambda=power<

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_funcattrs.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_funcattrs.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_funcattrs.py	Fri Oct 15 01:00:00 2010
@@ -7,6 +7,8 @@
 
 
 class FixFuncattrs(fixer_base.BaseFix):
+    BM_compatible = True
+
     PATTERN = """
     power< any+ trailer< '.' attr=('func_closure' | 'func_doc' | 'func_globals'
                                   | 'func_name' | 'func_defaults' | 'func_code'

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_future.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_future.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_future.py	Fri Oct 15 01:00:00 2010
@@ -9,6 +9,8 @@
 from ..fixer_util import BlankLine
 
 class FixFuture(fixer_base.BaseFix):
+    BM_compatible = True
+
     PATTERN = """import_from< 'from' module_name="__future__" 'import' any >"""
 
     # This should be run last -- some things check for the import

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_getcwdu.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_getcwdu.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_getcwdu.py	Fri Oct 15 01:00:00 2010
@@ -8,6 +8,7 @@
 from ..fixer_util import Name
 
 class FixGetcwdu(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
               power< 'os' trailer< dot='.' name='getcwdu' > any* >

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_has_key.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_has_key.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_has_key.py	Fri Oct 15 01:00:00 2010
@@ -37,6 +37,7 @@
 
 
 class FixHasKey(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
     anchor=power<

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_idioms.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_idioms.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_idioms.py	Fri Oct 15 01:00:00 2010
@@ -35,7 +35,6 @@
 TYPE = "power< 'type' trailer< '(' x=any ')' > >"
 
 class FixIdioms(fixer_base.BaseFix):
-
     explicit = True # The user must ask for this fixer
 
     PATTERN = r"""

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_import.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_import.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_import.py	Fri Oct 15 01:00:00 2010
@@ -36,6 +36,7 @@
 
 
 class FixImport(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
     import_from< 'from' imp=any 'import' ['('] any [')'] >

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_imports.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_imports.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_imports.py	Fri Oct 15 01:00:00 2010
@@ -84,6 +84,8 @@
 
 class FixImports(fixer_base.BaseFix):
 
+    BM_compatible = True
+    keep_line_order = True
     # This is overridden in fix_imports2.
     mapping = MAPPING
 

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_input.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_input.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_input.py	Fri Oct 15 01:00:00 2010
@@ -11,7 +11,7 @@
 
 
 class FixInput(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = """
               power< 'input' args=trailer< '(' [any] ')' > >
               """

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_intern.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_intern.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_intern.py	Fri Oct 15 01:00:00 2010
@@ -12,6 +12,8 @@
 
 
 class FixIntern(fixer_base.BaseFix):
+    BM_compatible = True
+    order = "pre"
 
     PATTERN = """
     power< 'intern'

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_isinstance.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_isinstance.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_isinstance.py	Fri Oct 15 01:00:00 2010
@@ -14,7 +14,7 @@
 
 
 class FixIsinstance(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = """
     power<
         'isinstance'

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools.py	Fri Oct 15 01:00:00 2010
@@ -12,6 +12,7 @@
 from ..fixer_util import Name
 
 class FixItertools(fixer_base.BaseFix):
+    BM_compatible = True
     it_funcs = "('imap'|'ifilter'|'izip'|'ifilterfalse')"
     PATTERN = """
               power< it='itertools'

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools_imports.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools_imports.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_itertools_imports.py	Fri Oct 15 01:00:00 2010
@@ -6,6 +6,7 @@
 
 
 class FixItertoolsImports(fixer_base.BaseFix):
+    BM_compatible = True
     PATTERN = """
               import_from< 'from' 'itertools' 'import' imports=any >
               """ %(locals())

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_long.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_long.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_long.py	Fri Oct 15 01:00:00 2010
@@ -10,7 +10,7 @@
 
 
 class FixLong(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = "'long'"
 
     def transform(self, node, results):

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_map.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_map.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_map.py	Fri Oct 15 01:00:00 2010
@@ -26,6 +26,7 @@
 from ..pygram import python_symbols as syms
 
 class FixMap(fixer_base.ConditionalFix):
+    BM_compatible = True
 
     PATTERN = """
     map_none=power<

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_metaclass.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_metaclass.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_metaclass.py	Fri Oct 15 01:00:00 2010
@@ -143,6 +143,7 @@
 
 
 class FixMetaclass(fixer_base.BaseFix):
+    BM_compatible = True
 
     PATTERN = """
     classdef<any*>

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_methodattrs.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_methodattrs.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_methodattrs.py	Fri Oct 15 01:00:00 2010
@@ -13,6 +13,7 @@
     }
 
 class FixMethodattrs(fixer_base.BaseFix):
+    BM_compatible = True
     PATTERN = """
     power< any+ trailer< '.' attr=('im_func' | 'im_self' | 'im_class') > any* >
     """

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_next.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_next.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_next.py	Fri Oct 15 01:00:00 2010
@@ -15,6 +15,7 @@
 
 
 class FixNext(fixer_base.BaseFix):
+    BM_compatible = True
     PATTERN = """
     power< base=any+ trailer< '.' attr='next' > trailer< '(' ')' > >
     |

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_nonzero.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_nonzero.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_nonzero.py	Fri Oct 15 01:00:00 2010
@@ -6,6 +6,7 @@
 from ..fixer_util import Name, syms
 
 class FixNonzero(fixer_base.BaseFix):
+    BM_compatible = True
     PATTERN = """
     classdef< 'class' any+ ':'
               suite< any*

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_operator.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_operator.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_operator.py	Fri Oct 15 01:00:00 2010
@@ -14,7 +14,16 @@
 from lib2to3.fixer_util import Call, Name, String, touch_import
 
 
+def invocation(s):
+    def dec(f):
+        f.invocation = s
+        return f
+    return dec
+
+
 class FixOperator(fixer_base.BaseFix):
+    BM_compatible = True
+    order = "pre"
 
     methods = """
               method=('isCallable'|'sequenceIncludes'
@@ -34,34 +43,34 @@
         if method is not None:
             return method(node, results)
 
+    @invocation("operator.contains(%s)")
     def _sequenceIncludes(self, node, results):
-        """operator.contains(%s)"""
         return self._handle_rename(node, results, u"contains")
 
+    @invocation("hasattr(%s, '__call__')")
     def _isCallable(self, node, results):
-        """hasattr(%s, '__call__')"""
         obj = results["obj"]
         args = [obj.clone(), String(u", "), String(u"'__call__'")]
         return Call(Name(u"hasattr"), args, prefix=node.prefix)
 
+    @invocation("operator.mul(%s)")
     def _repeat(self, node, results):
-        """operator.mul(%s)"""
         return self._handle_rename(node, results, u"mul")
 
+    @invocation("operator.imul(%s)")
     def _irepeat(self, node, results):
-        """operator.imul(%s)"""
         return self._handle_rename(node, results, u"imul")
 
+    @invocation("isinstance(%s, collections.Sequence)")
     def _isSequenceType(self, node, results):
-        """isinstance(%s, collections.Sequence)"""
         return self._handle_type2abc(node, results, u"collections", u"Sequence")
 
+    @invocation("isinstance(%s, collections.Mapping)")
     def _isMappingType(self, node, results):
-        """isinstance(%s, collections.Mapping)"""
         return self._handle_type2abc(node, results, u"collections", u"Mapping")
 
+    @invocation("isinstance(%s, numbers.Number)")
     def _isNumberType(self, node, results):
-        """isinstance(%s, numbers.Number)"""
         return self._handle_type2abc(node, results, u"numbers", u"Number")
 
     def _handle_rename(self, node, results, name):
@@ -82,6 +91,6 @@
                 return method
             else:
                 sub = (unicode(results["obj"]),)
-                invocation_str = unicode(method.__doc__) % sub
+                invocation_str = unicode(method.invocation) % sub
                 self.warning(node, u"You should use '%s' here." % invocation_str)
         return None

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_paren.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_paren.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_paren.py	Fri Oct 15 01:00:00 2010
@@ -10,6 +10,8 @@
 
 # XXX This doesn't support nested for loops like [x for x in 1, 2 for x in 1, 2]
 class FixParen(fixer_base.BaseFix):
+    BM_compatible = True
+
     PATTERN = """
         atom< ('[' | '(')
             (listmaker< any

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_print.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_print.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_print.py	Fri Oct 15 01:00:00 2010
@@ -28,6 +28,8 @@
 
 class FixPrint(fixer_base.BaseFix):
 
+    BM_compatible = True
+
     PATTERN = """
               simple_stmt< any* bare='print' any* > | print_stmt
               """

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_raise.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_raise.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_raise.py	Fri Oct 15 01:00:00 2010
@@ -4,6 +4,7 @@
 raise E       -> raise E
 raise E, V    -> raise E(V)
 raise E, V, T -> raise E(V).with_traceback(T)
+raise E, None, T -> raise E.with_traceback(T)
 
 raise (((E, E'), E''), E'''), V -> raise E(V)
 raise "foo", V, T               -> warns about string exceptions
@@ -29,6 +30,7 @@
 
 class FixRaise(fixer_base.BaseFix):
 
+    BM_compatible = True
     PATTERN = """
     raise_stmt< 'raise' exc=any [',' val=any [',' tb=any]] >
     """
@@ -37,8 +39,9 @@
         syms = self.syms
 
         exc = results["exc"].clone()
-        if exc.type is token.STRING:
-            self.cannot_convert(node, "Python 3 does not support string exceptions")
+        if exc.type == token.STRING:
+            msg = "Python 3 does not support string exceptions"
+            self.cannot_convert(node, msg)
             return
 
         # Python 2 supports
@@ -52,7 +55,7 @@
                 # exc.children[1:-1] is the unparenthesized tuple
                 # exc.children[1].children[0] is the first element of the tuple
                 exc = exc.children[1].children[0].clone()
-            exc.prefix = " "
+            exc.prefix = u" "
 
         if "val" not in results:
             # One-argument raise
@@ -71,7 +74,12 @@
             tb = results["tb"].clone()
             tb.prefix = u""
 
-            e = Call(exc, args)
+            e = exc
+            # If there's a traceback and None is passed as the value, then don't
+            # add a call, since the user probably just wants to add a
+            # traceback. See issue #9661.
+            if val.type != token.NAME or val.value != u"None":
+                e = Call(exc, args)
             with_tb = Attr(e, Name(u'with_traceback')) + [ArgList([tb])]
             new = pytree.Node(syms.simple_stmt, [Name(u"raise")] + with_tb)
             new.prefix = node.prefix

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_raw_input.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_raw_input.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_raw_input.py	Fri Oct 15 01:00:00 2010
@@ -7,6 +7,7 @@
 
 class FixRawInput(fixer_base.BaseFix):
 
+    BM_compatible = True
     PATTERN = """
               power< name='raw_input' trailer< '(' [any] ')' > any* >
               """

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_reduce.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_reduce.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_reduce.py	Fri Oct 15 01:00:00 2010
@@ -14,6 +14,9 @@
 
 class FixReduce(fixer_base.BaseFix):
 
+    BM_compatible = True
+    order = "pre"
+
     PATTERN = """
     power< 'reduce'
         trailer< '('

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_renames.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_renames.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_renames.py	Fri Oct 15 01:00:00 2010
@@ -40,6 +40,7 @@
 
 
 class FixRenames(fixer_base.BaseFix):
+    BM_compatible = True
     PATTERN = "|".join(build_pattern())
 
     order = "pre" # Pre-order tree traversal

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_repr.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_repr.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_repr.py	Fri Oct 15 01:00:00 2010
@@ -10,6 +10,7 @@
 
 class FixRepr(fixer_base.BaseFix):
 
+    BM_compatible = True
     PATTERN = """
               atom < '`' expr=any '`' >
               """

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_set_literal.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_set_literal.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_set_literal.py	Fri Oct 15 01:00:00 2010
@@ -11,6 +11,7 @@
 
 class FixSetLiteral(fixer_base.BaseFix):
 
+    BM_compatible = True
     explicit = True
 
     PATTERN = """power< 'set' trailer< '('

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_standarderror.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_standarderror.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_standarderror.py	Fri Oct 15 01:00:00 2010
@@ -9,7 +9,7 @@
 
 
 class FixStandarderror(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = """
               'StandardError'
               """

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_sys_exc.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_sys_exc.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_sys_exc.py	Fri Oct 15 01:00:00 2010
@@ -14,6 +14,7 @@
 class FixSysExc(fixer_base.BaseFix):
     # This order matches the ordering of sys.exc_info().
     exc_info = [u"exc_type", u"exc_value", u"exc_traceback"]
+    BM_compatible = True
     PATTERN = """
               power< 'sys' trailer< dot='.' attribute=(%s) > >
               """ % '|'.join("'%s'" % e for e in exc_info)

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_throw.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_throw.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_throw.py	Fri Oct 15 01:00:00 2010
@@ -14,7 +14,7 @@
 from ..fixer_util import Name, Call, ArgList, Attr, is_tuple
 
 class FixThrow(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = """
     power< any trailer< '.' 'throw' >
            trailer< '(' args=arglist< exc=any ',' val=any [',' tb=any] > ')' >

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_tuple_params.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_tuple_params.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_tuple_params.py	Fri Oct 15 01:00:00 2010
@@ -29,6 +29,10 @@
            stmt.children[0].type == token.STRING
 
 class FixTupleParams(fixer_base.BaseFix):
+    run_order = 4 #use a lower order since lambda is part of other
+                  #patterns
+    BM_compatible = True
+
     PATTERN = """
               funcdef< 'def' any parameters< '(' args=any ')' >
                        ['->' any] ':' suite=any+ >

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_types.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_types.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_types.py	Fri Oct 15 01:00:00 2010
@@ -52,7 +52,7 @@
 _pats = ["power< 'types' trailer< '.' name='%s' > >" % t for t in _TYPE_MAPPING]
 
 class FixTypes(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = '|'.join(_pats)
 
     def transform(self, node, results):

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_unicode.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_unicode.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_unicode.py	Fri Oct 15 01:00:00 2010
@@ -10,7 +10,7 @@
 _literal_re = re.compile(ur"[uU][rR]?[\'\"]")
 
 class FixUnicode(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = "STRING | 'unicode' | 'unichr'"
 
     def transform(self, node, results):

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_urllib.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_urllib.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_urllib.py	Fri Oct 15 01:00:00 2010
@@ -8,7 +8,7 @@
 from lib2to3.fixes.fix_imports import alternates, FixImports
 from lib2to3 import fixer_base
 from lib2to3.fixer_util import (Name, Comma, FromImport, Newline,
-                                find_indentation)
+                                find_indentation, Node, syms)
 
 MAPPING = {"urllib":  [
                 ("urllib.request",
@@ -121,26 +121,37 @@
             mod_dict = {}
             members = results["members"]
             for member in members:
-                member = member.value
                 # we only care about the actual members
-                if member != u",":
+                if member.type == syms.import_as_name:
+                    as_name = member.children[2].value
+                    member_name = member.children[0].value
+                else:
+                    member_name = member.value
+                    as_name = None
+                if member_name != u",":
                     for change in MAPPING[mod_member.value]:
-                        if member in change[1]:
-                            if change[0] in mod_dict:
-                                mod_dict[change[0]].append(member)
-                            else:
-                                mod_dict[change[0]] = [member]
+                        if member_name in change[1]:
+                            if change[0] not in mod_dict:
                                 modules.append(change[0])
+                            mod_dict.setdefault(change[0], []).append(member)
 
             new_nodes = []
             indentation = find_indentation(node)
             first = True
+            def handle_name(name, prefix):
+                if name.type == syms.import_as_name:
+                    kids = [Name(name.children[0].value, prefix=prefix),
+                            name.children[1].clone(),
+                            name.children[2].clone()]
+                    return [Node(syms.import_as_name, kids)]
+                return [Name(name.value, prefix=prefix)]
             for module in modules:
                 elts = mod_dict[module]
                 names = []
                 for elt in elts[:-1]:
-                    names.extend([Name(elt, prefix=pref), Comma()])
-                names.append(Name(elts[-1], prefix=pref))
+                    names.extend(handle_name(elt, pref))
+                    names.append(Comma())
+                names.extend(handle_name(elts[-1], pref))
                 new = FromImport(module, names)
                 if not first or node.parent.prefix.endswith(indentation):
                     new.prefix = indentation

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_xrange.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_xrange.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_xrange.py	Fri Oct 15 01:00:00 2010
@@ -10,7 +10,7 @@
 
 
 class FixXrange(fixer_base.BaseFix):
-
+    BM_compatible = True
     PATTERN = """
               power<
                  (name='range'|name='xrange') trailer< '(' args=any ')' >

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_xreadlines.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_xreadlines.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_xreadlines.py	Fri Oct 15 01:00:00 2010
@@ -9,6 +9,7 @@
 
 
 class FixXreadlines(fixer_base.BaseFix):
+    BM_compatible = True
     PATTERN = """
     power< call=any+ trailer< '.' 'xreadlines' > trailer< '(' ')' > >
     |

Modified: python/branches/release27-maint/Lib/lib2to3/fixes/fix_zip.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/fixes/fix_zip.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/fixes/fix_zip.py	Fri Oct 15 01:00:00 2010
@@ -13,6 +13,7 @@
 
 class FixZip(fixer_base.ConditionalFix):
 
+    BM_compatible = True
     PATTERN = """
     power< 'zip' args=trailer< '(' [any] ')' >
     >

Modified: python/branches/release27-maint/Lib/lib2to3/patcomp.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/patcomp.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/patcomp.py	Fri Oct 15 01:00:00 2010
@@ -52,14 +52,17 @@
         self.pysyms = pygram.python_symbols
         self.driver = driver.Driver(self.grammar, convert=pattern_convert)
 
-    def compile_pattern(self, input, debug=False):
+    def compile_pattern(self, input, debug=False, with_tree=False):
         """Compiles a pattern string to a nested pytree.*Pattern object."""
         tokens = tokenize_wrapper(input)
         try:
             root = self.driver.parse_tokens(tokens, debug=debug)
-        except parse.ParseError, e:
+        except parse.ParseError as e:
             raise PatternSyntaxError(str(e))
-        return self.compile_node(root)
+        if with_tree:
+            return self.compile_node(root), root
+        else:
+            return self.compile_node(root)
 
     def compile_node(self, node):
         """Compiles a node, recursively.

Modified: python/branches/release27-maint/Lib/lib2to3/pygram.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/pygram.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/pygram.py	Fri Oct 15 01:00:00 2010
@@ -13,6 +13,8 @@
 
 # The grammar file
 _GRAMMAR_FILE = os.path.join(os.path.dirname(__file__), "Grammar.txt")
+_PATTERN_GRAMMAR_FILE = os.path.join(os.path.dirname(__file__),
+                                     "PatternGrammar.txt")
 
 
 class Symbols(object):
@@ -33,3 +35,6 @@
 
 python_grammar_no_print_statement = python_grammar.copy()
 del python_grammar_no_print_statement.keywords["print"]
+
+pattern_grammar = driver.load_grammar(_PATTERN_GRAMMAR_FILE)
+pattern_symbols = Symbols(pattern_grammar)

Modified: python/branches/release27-maint/Lib/lib2to3/pytree.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/pytree.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/pytree.py	Fri Oct 15 01:00:00 2010
@@ -16,7 +16,6 @@
 import warnings
 from StringIO import StringIO
 
-
 HUGE = 0x7FFFFFFF  # maximum repeat count, default max
 
 _type_reprs = {}
@@ -30,7 +29,6 @@
             if type(val) == int: _type_reprs[val] = name
     return _type_reprs.setdefault(type_num, type_num)
 
-
 class Base(object):
 
     """
@@ -47,6 +45,7 @@
     parent = None  # Parent node pointer, or None
     children = ()  # Tuple of subnodes
     was_changed = False
+    was_checked = False
 
     def __new__(cls, *args, **kwds):
         """Constructor that prevents Base from being instantiated."""
@@ -213,6 +212,16 @@
                     return None
                 return self.parent.children[i-1]
 
+    def leaves(self):
+        for child in self.children:
+            for x in child.leaves():
+                yield x
+
+    def depth(self):
+        if self.parent is None:
+            return 0
+        return 1 + self.parent.depth()
+
     def get_suffix(self):
         """
         Return the string immediately following the invocant node. This is
@@ -227,12 +236,14 @@
         def __str__(self):
             return unicode(self).encode("ascii")
 
-
 class Node(Base):
 
     """Concrete implementation for interior nodes."""
 
-    def __init__(self, type, children, context=None, prefix=None):
+    def __init__(self,type, children,
+                 context=None,
+                 prefix=None,
+                 fixers_applied=None):
         """
         Initializer.
 
@@ -249,6 +260,10 @@
             ch.parent = self
         if prefix is not None:
             self.prefix = prefix
+        if fixers_applied:
+            self.fixers_applied = fixers_applied[:]
+        else:
+            self.fixers_applied = None
 
     def __repr__(self):
         """Return a canonical string representation."""
@@ -273,7 +288,8 @@
 
     def clone(self):
         """Return a cloned (deep) copy of self."""
-        return Node(self.type, [ch.clone() for ch in self.children])
+        return Node(self.type, [ch.clone() for ch in self.children],
+                    fixers_applied=self.fixers_applied)
 
     def post_order(self):
         """Return a post-order iterator for the tree."""
@@ -341,7 +357,10 @@
     lineno = 0    # Line where this token starts in the input
     column = 0    # Column where this token tarts in the input
 
-    def __init__(self, type, value, context=None, prefix=None):
+    def __init__(self, type, value,
+                 context=None,
+                 prefix=None,
+                 fixers_applied=[]):
         """
         Initializer.
 
@@ -355,6 +374,7 @@
         self.value = value
         if prefix is not None:
             self._prefix = prefix
+        self.fixers_applied = fixers_applied[:]
 
     def __repr__(self):
         """Return a canonical string representation."""
@@ -380,7 +400,11 @@
     def clone(self):
         """Return a cloned (deep) copy of self."""
         return Leaf(self.type, self.value,
-                    (self.prefix, (self.lineno, self.column)))
+                    (self.prefix, (self.lineno, self.column)),
+                    fixers_applied=self.fixers_applied)
+
+    def leaves(self):
+        yield self
 
     def post_order(self):
         """Return a post-order iterator for the tree."""

Modified: python/branches/release27-maint/Lib/lib2to3/refactor.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/refactor.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/refactor.py	Fri Oct 15 01:00:00 2010
@@ -24,7 +24,10 @@
 
 # Local imports
 from .pgen2 import driver, tokenize, token
+from .fixer_util import find_root
 from . import pytree, pygram
+from . import btm_utils as bu
+from . import btm_matcher as bm
 
 
 def get_all_fix_names(fixer_pkg, remove_prefix=True):
@@ -201,11 +204,28 @@
                                     logger=self.logger)
         self.pre_order, self.post_order = self.get_fixers()
 
-        self.pre_order_heads = _get_headnode_dict(self.pre_order)
-        self.post_order_heads = _get_headnode_dict(self.post_order)
 
         self.files = []  # List of files that were or should be modified
 
+        self.BM = bm.BottomMatcher()
+        self.bmi_pre_order = [] # Bottom Matcher incompatible fixers
+        self.bmi_post_order = []
+
+        for fixer in chain(self.post_order, self.pre_order):
+            if fixer.BM_compatible:
+                self.BM.add_fixer(fixer)
+                # remove fixers that will be handled by the bottom-up
+                # matcher
+            elif fixer in self.pre_order:
+                self.bmi_pre_order.append(fixer)
+            elif fixer in self.post_order:
+                self.bmi_post_order.append(fixer)
+
+        self.bmi_pre_order_heads = _get_headnode_dict(self.bmi_pre_order)
+        self.bmi_post_order_heads = _get_headnode_dict(self.bmi_post_order)
+
+
+
     def get_fixers(self):
         """Inspects the options to load the requested patterns and handlers.
 
@@ -268,6 +288,7 @@
 
     def refactor(self, items, write=False, doctests_only=False):
         """Refactor a list of files and directories."""
+
         for dir_or_file in items:
             if os.path.isdir(dir_or_file):
                 self.refactor_dir(dir_or_file, write, doctests_only)
@@ -299,7 +320,7 @@
         """
         try:
             f = open(filename, "rb")
-        except IOError, err:
+        except IOError as err:
             self.log_error("Can't open %s: %s", filename, err)
             return None, None
         try:
@@ -348,7 +369,7 @@
             self.driver.grammar = pygram.python_grammar_no_print_statement
         try:
             tree = self.driver.parse_string(data)
-        except Exception, err:
+        except Exception as err:
             self.log_error("Can't parse %s: %s: %s",
                            name, err.__class__.__name__, err)
             return
@@ -378,6 +399,10 @@
     def refactor_tree(self, tree, name):
         """Refactors a parse tree (modifying the tree in place).
 
+        For compatible patterns the bottom matcher module is
+        used. Otherwise the tree is traversed node-to-node for
+        matches.
+
         Args:
             tree: a pytree.Node instance representing the root of the tree
                   to be refactored.
@@ -386,11 +411,65 @@
         Returns:
             True if the tree was modified, False otherwise.
         """
+
         for fixer in chain(self.pre_order, self.post_order):
             fixer.start_tree(tree, name)
 
-        self.traverse_by(self.pre_order_heads, tree.pre_order())
-        self.traverse_by(self.post_order_heads, tree.post_order())
+        #use traditional matching for the incompatible fixers
+        self.traverse_by(self.bmi_pre_order_heads, tree.pre_order())
+        self.traverse_by(self.bmi_post_order_heads, tree.post_order())
+
+        # obtain a set of candidate nodes
+        match_set = self.BM.run(tree.leaves())
+
+        while any(match_set.values()):
+            for fixer in self.BM.fixers:
+                if fixer in match_set and match_set[fixer]:
+                    #sort by depth; apply fixers from bottom(of the AST) to top
+                    match_set[fixer].sort(key=pytree.Base.depth, reverse=True)
+
+                    if fixer.keep_line_order:
+                        #some fixers(eg fix_imports) must be applied
+                        #with the original file's line order
+                        match_set[fixer].sort(key=pytree.Base.get_lineno)
+
+                    for node in list(match_set[fixer]):
+                        if node in match_set[fixer]:
+                            match_set[fixer].remove(node)
+
+                        try:
+                            find_root(node)
+                        except AssertionError:
+                            # this node has been cut off from a
+                            # previous transformation ; skip
+                            continue
+
+                        if node.fixers_applied and fixer in node.fixers_applied:
+                            # do not apply the same fixer again
+                            continue
+
+                        results = fixer.match(node)
+
+                        if results:
+                            new = fixer.transform(node, results)
+                            if new is not None:
+                                node.replace(new)
+                                #new.fixers_applied.append(fixer)
+                                for node in new.post_order():
+                                    # do not apply the fixer again to
+                                    # this or any subnode
+                                    if not node.fixers_applied:
+                                        node.fixers_applied = []
+                                    node.fixers_applied.append(fixer)
+
+                                # update the original match set for
+                                # the added code
+                                new_matches = self.BM.run(new.leaves())
+                                for fxr in new_matches:
+                                    if not fxr in match_set:
+                                        match_set[fxr]=[]
+
+                                    match_set[fxr].extend(new_matches[fxr])
 
         for fixer in chain(self.pre_order, self.post_order):
             fixer.finish_tree(tree, name)
@@ -448,12 +527,12 @@
         """
         try:
             f = _open_with_encoding(filename, "w", encoding=encoding)
-        except os.error, err:
+        except os.error as err:
             self.log_error("Can't create %s: %s", filename, err)
             return
         try:
             f.write(_to_system_newlines(new_text))
-        except os.error, err:
+        except os.error as err:
             self.log_error("Can't write %s: %s", filename, err)
         finally:
             f.close()
@@ -516,7 +595,7 @@
         """
         try:
             tree = self.parse_block(block, lineno, indent)
-        except Exception, err:
+        except Exception as err:
             if self.logger.isEnabledFor(logging.DEBUG):
                 for line in block:
                     self.log_debug("Source: %s", line.rstrip(u"\n"))

Modified: python/branches/release27-maint/Lib/lib2to3/tests/data/bom.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/tests/data/bom.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/tests/data/bom.py	Fri Oct 15 01:00:00 2010
@@ -1,3 +1,2 @@
 # coding: utf-8
 print "BOM BOOM!"
-

Modified: python/branches/release27-maint/Lib/lib2to3/tests/test_fixers.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/tests/test_fixers.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/tests/test_fixers.py	Fri Oct 15 01:00:00 2010
@@ -868,6 +868,11 @@
                     raise Exception(5).with_traceback(6) # foo"""
         self.check(b, a)
 
+    def test_None_value(self):
+        b = """raise Exception(5), None, tb"""
+        a = """raise Exception(5).with_traceback(tb)"""
+        self.check(b, a)
+
     def test_tuple_value(self):
         b = """raise Exception, (5, 6, 7)"""
         a = """raise Exception(5, 6, 7)"""
@@ -1812,6 +1817,9 @@
                     b = "from %s import %s as foo_bar" % (old, member)
                     a = "from %s import %s as foo_bar" % (new, member)
                     self.check(b, a)
+                    b = "from %s import %s as blah, %s" % (old, member, member)
+                    a = "from %s import %s as blah, %s" % (new, member, member)
+                    self.check(b, a)
 
     def test_star(self):
         for old in self.modules:

Modified: python/branches/release27-maint/Lib/lib2to3/tests/test_pytree.py
==============================================================================
--- python/branches/release27-maint/Lib/lib2to3/tests/test_pytree.py	(original)
+++ python/branches/release27-maint/Lib/lib2to3/tests/test_pytree.py	Fri Oct 15 01:00:00 2010
@@ -178,6 +178,27 @@
         self.assertEqual(str(n1), "foo**bar")
         self.assertTrue(isinstance(n1.children, list))
 
+    def test_leaves(self):
+        l1 = pytree.Leaf(100, "foo")
+        l2 = pytree.Leaf(100, "bar")
+        l3 = pytree.Leaf(100, "fooey")
+        n2 = pytree.Node(1000, [l1, l2])
+        n3 = pytree.Node(1000, [l3])
+        n1 = pytree.Node(1000, [n2, n3])
+
+        self.assertEqual(list(n1.leaves()), [l1, l2, l3])
+
+    def test_depth(self):
+        l1 = pytree.Leaf(100, "foo")
+        l2 = pytree.Leaf(100, "bar")
+        n2 = pytree.Node(1000, [l1, l2])
+        n3 = pytree.Node(1000, [])
+        n1 = pytree.Node(1000, [n2, n3])
+
+        self.assertEqual(l1.depth(), 2)
+        self.assertEqual(n3.depth(), 1)
+        self.assertEqual(n1.depth(), 0)
+
     def test_post_order(self):
         l1 = pytree.Leaf(100, "foo")
         l2 = pytree.Leaf(100, "bar")
