[Python-porting] bugs in 2to3 tools when porting sympy

[Ondrej Certik]

Hi,

here is a list of bugs that are triggered by running the 2to3 tool on
sympy, together with steps how to reproduce them. I am running 2to3
tool from python3.1. I created a "2to3-bugs" branch on my github
account with sympy and I am going to leave that branch there without
further modificaitons, so that you can easily debug it anytime.

1) fails with unicode:

$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 . > p
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Can't parse ./data/IPython/ipythonrc-sympy:
ParseError: bad input: type=1, value='ipythonrc', context=(' ', (25,
8))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/BasicGAtest.py: ParseError: bad input:
type=5, value='        ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/conformalgeometryGAtest.py: ParseError:
bad input: type=5, value='        ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/headerGAtest.py: ParseError: bad input:
type=0, value='', context=('\n', (26, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/hyperbolicGAtest.py: ParseError: bad
input: type=5, value='        ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/reciprocalframeGAtest.py: ParseError:
bad input: type=5, value='        ', context=('', (1, 0))
Traceback (most recent call last):
 File "/home/ondrej/ext/Python-3.1/Tools/scripts/2to3", line 6, in <module>
   sys.exit(main("lib2to3.fixes"))
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 132, in main
   options.processes)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
544, in refactor
   items, write, doctests_only)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
207, in refactor
   self.refactor_dir(dir_or_file, write, doctests_only)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
225, in refactor_dir
   self.refactor_file(fullname, write, doctests_only)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
584, in refactor_file
   *args, **kwargs)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
264, in refactor_file
   write=write, encoding=encoding)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
363, in processed_file
   self.print_output(diff_texts(old_text, new_text, filename))
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 47, in
print_output
   print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\u03b1' in
position 32: ordinal not in range(128)


this was already reported here: http://bugs.python.org/issue5093

it also reports the "RefactoringTool: Can't parse" errors, I don't
know what that means.

2) It doesn't fix all imports:

$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 -w .    # this is necessary due to the bug 1)

now if you look into the sympy/__init__.py file, you need to apply the
following patch:

-from series import *
-from functions import *
-from ntheory import *
-from concrete import *
-from simplify import *
-from solvers import *
-from matrices import *
-from geometry import *
-from utilities import *
-from integrals import *
+from .series import *
+from .functions import *
+from .ntheory import *
+from .concrete import *
+from .simplify import *
+from .solvers import *
+from .matrices import *
+from .geometry import *
+from .utilities import *
+from .integrals import *


the same about most of other __init__.py files in sympy. I believe
this is a bug in 2to3 tool, that should do it automatically.

3) there is a file sympy/printing/repr.py which is imported in
sympy/printing/__init__.py.  For some reason the 2to3 tool produces a
patch:

-from repr import srepr
+from reprlib import srepr

however the correct patch is:

-from repr import srepr
+from .repr import srepr

I think 2to3 tool should first check if a local file is present and if
so, treat this as a local import, not a std library import.


Besides the 3 bugs above, there are lots of small annoyances, like
that sys.version_info has changed, so one has to use tuple() around
it, that .sort() doesn't accept a cmp() like callback anymore, and
similar things, which hopefully should be relatively easy to fix in a
manner that works in all pythons.

Then there is a big problem that for some reason, hash() doesn't work
on our Integer class anymore, but seems to work on some other classes.
After I fixed the above errors by hand, here is a result of our test
suite (and this is only sympy/core, e.g. about 1/10 of all our
tests...):

http://groups.google.com/group/sympy/msg/525e8420ebb2b08d

so it is now clear that it will be a lot of work to port everything.
But I still believe it would be possible to just maintain one
codebase, and run 2to3 tool to produce python3 compatible library.

If anyone could give me some help with the bugs in 2to3 tool, it'd be
awesome, it would simplify our job a lot. Otherwise we would have to
create some specialized scripts to convert the rest of sympy (after
the 2to3 tool run) to be python3 compatible.

Ondrej