[Python-porting] bugs in 2to3 tools when porting sympy

[Ondrej Certik]

On Sun, Jul 5, 2009 at 6:21 PM, Benjamin Peterson<benjamin at python.org> wrote:
> Could you try with the latest version of 2to3 from the sandbox? [1]
> I've recently fixed I think all of the bugs you've described.

I tested the bug 1):

$ python2.6 ~/repos/2to3/2to3 . > p
WARNING: not writing files and not printing diffs; that's not very useful
root: Generating grammar tables from
/home/ondrej/repos/2to3/lib2to3/PatternGrammar.txt
root: Writing grammar tables to
/home/ondrej/repos/2to3/lib2to3/PatternGrammar2.6.2.final.0.pickle
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored ./setup.py
RefactoringTool: Refactored ./setupegg.py
RefactoringTool: Refactored ./bin/adapt_paths.py
RefactoringTool: Refactored ./bin/coverage_report.py
RefactoringTool: Refactored ./bin/generate_test_list.py
RefactoringTool: Refactored ./bin/isympy
RefactoringTool: Refactored ./bin/sympy_time.py
RefactoringTool: Refactored ./bin/sympy_time_cache.py
RefactoringTool: Refactored ./bin/test_import.py
RefactoringTool: Can't parse ./data/IPython/ipythonrc-sympy:
ParseError: bad input: type=1, value=u'ipythonrc', context=(u' ', (25,
8))
RefactoringTool: Refactored ./doc/generate_reference.py
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/BasicGAtest.py: ParseError: bad input:
type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Refactored ./doc/src/modules/galgebra/GA/Dirac.py
RefactoringTool: Refactored ./doc/src/modules/galgebra/GA/Maxwell.py
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/conformalgeometryGAtest.py: ParseError:
bad input: type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Refactored ./doc/src/modules/galgebra/GA/coords.py
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/headerGAtest.py: ParseError: bad input:
type=0, value='', context=(u'\n', (26, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/hyperbolicGAtest.py: ParseError: bad
input: type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/reciprocalframeGAtest.py: ParseError:
bad input: type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Refactored ./doc/src/modules/galgebra/latex_ex/Maxwell.py
RefactoringTool: Refactored ./doc/src/modules/galgebra/latex_ex/latexdemo.py
RefactoringTool: Refactored ./examples/all.py
RefactoringTool: Refactored ./examples/advanced/curvilinear_coordinates.py
RefactoringTool: Refactored ./examples/advanced/fem.py
RefactoringTool: Refactored ./examples/advanced/gibbs_phenomenon.py
RefactoringTool: Refactored ./examples/advanced/pidigits.py
RefactoringTool: Refactored ./examples/advanced/plotting.py
RefactoringTool: Refactored ./examples/advanced/qft.py
RefactoringTool: Refactored ./examples/advanced/relativity.py
RefactoringTool: Refactored ./examples/beginner/basic.py
RefactoringTool: Refactored ./examples/beginner/differentiation.py
RefactoringTool: Refactored ./examples/beginner/expansion.py
RefactoringTool: Refactored ./examples/beginner/functions.py
RefactoringTool: Refactored ./examples/beginner/limits_examples.py
RefactoringTool: Refactored ./examples/beginner/precision.py
RefactoringTool: Refactored ./examples/beginner/print_pretty.py
RefactoringTool: Refactored ./examples/beginner/series.py
RefactoringTool: Refactored ./examples/beginner/substitution.py
RefactoringTool: Refactored ./examples/intermediate/differential_equations.py
RefactoringTool: No changes to ./examples/intermediate/mplot2d.py
RefactoringTool: Refactored ./examples/intermediate/trees.py
RefactoringTool: Refactored ./examples/intermediate/vandermonde.py
RefactoringTool: Refactored ./sympy/__init__.py
RefactoringTool: Refactored ./sympy/abc.py
RefactoringTool: Refactored ./sympy/benchmarks/bench_symbench.py
RefactoringTool: Refactored ./sympy/concrete/__init__.py
RefactoringTool: Refactored ./sympy/concrete/gosper.py
RefactoringTool: Refactored ./sympy/concrete/products.py
RefactoringTool: Refactored ./sympy/concrete/summations.py
RefactoringTool: Refactored ./sympy/concrete/sums_products.py
RefactoringTool: Refactored ./sympy/concrete/tests/test_sums_products.py
RefactoringTool: Refactored ./sympy/core/__init__.py
RefactoringTool: Refactored ./sympy/core/add.py
RefactoringTool: Refactored ./sympy/core/assumptions.py
RefactoringTool: Refactored ./sympy/core/ast_parser.py
RefactoringTool: Refactored ./sympy/core/ast_parser_python24.py
RefactoringTool: Refactored ./sympy/core/basic.py
RefactoringTool: Refactored ./sympy/core/cache.py
RefactoringTool: Refactored ./sympy/core/decorators.py
RefactoringTool: Refactored ./sympy/core/evalf.py
Traceback (most recent call last):
  File "/home/ondrej/repos/2to3/2to3", line 5, in <module>
    sys.exit(main("lib2to3.fixes"))
  File "/home/ondrej/repos/2to3/lib2to3/main.py", line 157, in main
    options.processes)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 545, in refactor
    items, write, doctests_only)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 207, in refactor
    self.refactor_dir(dir_or_file, write, doctests_only)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 225, in refactor_dir
    self.refactor_file(fullname, write, doctests_only)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 585, in refactor_file
    *args, **kwargs)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 260, in refactor_file
    tree = self.refactor_string(input, filename)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 286, in
refactor_string
    self.refactor_tree(tree, name)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 320, in refactor_tree
    self.traverse_by(self.post_order_heads, tree.post_order())
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 346, in traverse_by
    str(new) != str(node)):
  File "/home/ondrej/repos/2to3/lib2to3/pytree.py", line 226, in __str__
    return unicode(self).encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03b1' in
position 1248: ordinal not in range(128)



am I invoking it in the correct way? I just downloaded the svn
repository into ~/repos/2to3/.

I cannot easily test bug 2), because even this fails with the
UnicodeEncodeError:

python2.6 ~/repos/2to3/2to3 -w .

>
> [1] http://svn.python.org/projects/sandbox/trunk/2to3
>
>>
>> Besides the 3 bugs above, there are lots of small annoyances, like
>> that sys.version_info has changed, so one has to use tuple() around
>> it, that .sort() doesn't accept a cmp() like callback anymore, and
>> similar things, which hopefully should be relatively easy to fix in a
>> manner that works in all pythons.
>
> What are you doing with sys.version_info which makes it incompatible?

I had to apply this patch:

         v = sys.version_info
-        python_version = "%s.%s.%s-%s-%s" % v
+        python_version = "%s.%s.%s-%s-%s" % tuple(v)


the "-" line works in python3.0 and below, but stops working in
python3.1. The fix is trivial, let's not loose time with it. After
fixing the 2to3 tool, I have a serious problem that the __hash__
method is defined in the Basic() class, but in it's subclasses (Number
and Symbol):

Number.__hash__

returns None, but

Symbol.__hash__

correctly points to the Basic.hash method. As a result, Number is not
hashable and this breaks sympy. I already spent 2 hours debugging it,
so far the only clue is that __slots__ influence it, but so far I
wasn't able to isolate a simple failing testcase (if I create a simple
class hierarchy, it works in python3.1), the problem is that it's not
easy to delete just part of sympy -- it's either all or nothing, and
thus it's very difficult to figure out why python3 behaves as it
behaves.

If anyone on this list has experience with debugging such kind of a
problem in python 3, I would be interested in any input.

Ondrej