From ondrej at certik.cz  Mon Jul  6 00:22:04 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 5 Jul 2009 16:22:04 -0600
Subject: [Python-porting] bugs in 2to3 tools when porting sympy
Message-ID: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>

Hi,

here is a list of bugs that are triggered by running the 2to3 tool on
sympy, together with steps how to reproduce them. I am running 2to3
tool from python3.1. I created a "2to3-bugs" branch on my github
account with sympy and I am going to leave that branch there without
further modificaitons, so that you can easily debug it anytime.

1) fails with unicode:

$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 . > p
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Can't parse ./data/IPython/ipythonrc-sympy:
ParseError: bad input: type=1, value='ipythonrc', context=(' ', (25,
8))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/BasicGAtest.py: ParseError: bad input:
type=5, value='        ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/conformalgeometryGAtest.py: ParseError:
bad input: type=5, value='        ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/headerGAtest.py: ParseError: bad input:
type=0, value='', context=('\n', (26, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/hyperbolicGAtest.py: ParseError: bad
input: type=5, value='        ', context=('', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/reciprocalframeGAtest.py: ParseError:
bad input: type=5, value='        ', context=('', (1, 0))
Traceback (most recent call last):
 File "/home/ondrej/ext/Python-3.1/Tools/scripts/2to3", line 6, in <module>
   sys.exit(main("lib2to3.fixes"))
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 132, in main
   options.processes)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
544, in refactor
   items, write, doctests_only)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
207, in refactor
   self.refactor_dir(dir_or_file, write, doctests_only)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
225, in refactor_dir
   self.refactor_file(fullname, write, doctests_only)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
584, in refactor_file
   *args, **kwargs)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
264, in refactor_file
   write=write, encoding=encoding)
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/refactor.py", line
363, in processed_file
   self.print_output(diff_texts(old_text, new_text, filename))
 File "/home/ondrej/ext/Python-3.1/Lib/lib2to3/main.py", line 47, in
print_output
   print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\u03b1' in
position 32: ordinal not in range(128)


this was already reported here: http://bugs.python.org/issue5093

it also reports the "RefactoringTool: Can't parse" errors, I don't
know what that means.

2) It doesn't fix all imports:

$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b 2to3-bugs origin/2to3-bugs
$ 2to3 -w .    # this is necessary due to the bug 1)

now if you look into the sympy/__init__.py file, you need to apply the
following patch:

-from series import *
-from functions import *
-from ntheory import *
-from concrete import *
-from simplify import *
-from solvers import *
-from matrices import *
-from geometry import *
-from utilities import *
-from integrals import *
+from .series import *
+from .functions import *
+from .ntheory import *
+from .concrete import *
+from .simplify import *
+from .solvers import *
+from .matrices import *
+from .geometry import *
+from .utilities import *
+from .integrals import *


the same about most of other __init__.py files in sympy. I believe
this is a bug in 2to3 tool, that should do it automatically.

3) there is a file sympy/printing/repr.py which is imported in
sympy/printing/__init__.py.  For some reason the 2to3 tool produces a
patch:

-from repr import srepr
+from reprlib import srepr

however the correct patch is:

-from repr import srepr
+from .repr import srepr

I think 2to3 tool should first check if a local file is present and if
so, treat this as a local import, not a std library import.


Besides the 3 bugs above, there are lots of small annoyances, like
that sys.version_info has changed, so one has to use tuple() around
it, that .sort() doesn't accept a cmp() like callback anymore, and
similar things, which hopefully should be relatively easy to fix in a
manner that works in all pythons.

Then there is a big problem that for some reason, hash() doesn't work
on our Integer class anymore, but seems to work on some other classes.
After I fixed the above errors by hand, here is a result of our test
suite (and this is only sympy/core, e.g. about 1/10 of all our
tests...):

http://groups.google.com/group/sympy/msg/525e8420ebb2b08d

so it is now clear that it will be a lot of work to port everything.
But I still believe it would be possible to just maintain one
codebase, and run 2to3 tool to produce python3 compatible library.

If anyone could give me some help with the bugs in 2to3 tool, it'd be
awesome, it would simplify our job a lot. Otherwise we would have to
create some specialized scripts to convert the rest of sympy (after
the 2to3 tool run) to be python3 compatible.

Ondrej

From benjamin at python.org  Mon Jul  6 02:21:03 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 5 Jul 2009 19:21:03 -0500
Subject: [Python-porting] bugs in 2to3 tools when porting sympy
In-Reply-To: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>
References: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>
Message-ID: <1afaf6160907051721i5d85d44ew2399a40441e5b36b@mail.gmail.com>

Could you try with the latest version of 2to3 from the sandbox? [1]
I've recently fixed I think all of the bugs you've described.

[1] http://svn.python.org/projects/sandbox/trunk/2to3

>
> Besides the 3 bugs above, there are lots of small annoyances, like
> that sys.version_info has changed, so one has to use tuple() around
> it, that .sort() doesn't accept a cmp() like callback anymore, and
> similar things, which hopefully should be relatively easy to fix in a
> manner that works in all pythons.

What are you doing with sys.version_info which makes it incompatible?



-- 
Regards,
Benjamin

From ondrej at certik.cz  Mon Jul  6 02:54:46 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 5 Jul 2009 18:54:46 -0600
Subject: [Python-porting] bugs in 2to3 tools when porting sympy
In-Reply-To: <1afaf6160907051721i5d85d44ew2399a40441e5b36b@mail.gmail.com>
References: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>
	<1afaf6160907051721i5d85d44ew2399a40441e5b36b@mail.gmail.com>
Message-ID: <85b5c3130907051754y39a60b29s80475b77955c0fc4@mail.gmail.com>

On Sun, Jul 5, 2009 at 6:21 PM, Benjamin Peterson<benjamin at python.org> wrote:
> Could you try with the latest version of 2to3 from the sandbox? [1]
> I've recently fixed I think all of the bugs you've described.

I tested the bug 1):

$ python2.6 ~/repos/2to3/2to3 . > p
WARNING: not writing files and not printing diffs; that's not very useful
root: Generating grammar tables from
/home/ondrej/repos/2to3/lib2to3/PatternGrammar.txt
root: Writing grammar tables to
/home/ondrej/repos/2to3/lib2to3/PatternGrammar2.6.2.final.0.pickle
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored ./setup.py
RefactoringTool: Refactored ./setupegg.py
RefactoringTool: Refactored ./bin/adapt_paths.py
RefactoringTool: Refactored ./bin/coverage_report.py
RefactoringTool: Refactored ./bin/generate_test_list.py
RefactoringTool: Refactored ./bin/isympy
RefactoringTool: Refactored ./bin/sympy_time.py
RefactoringTool: Refactored ./bin/sympy_time_cache.py
RefactoringTool: Refactored ./bin/test_import.py
RefactoringTool: Can't parse ./data/IPython/ipythonrc-sympy:
ParseError: bad input: type=1, value=u'ipythonrc', context=(u' ', (25,
8))
RefactoringTool: Refactored ./doc/generate_reference.py
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/BasicGAtest.py: ParseError: bad input:
type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Refactored ./doc/src/modules/galgebra/GA/Dirac.py
RefactoringTool: Refactored ./doc/src/modules/galgebra/GA/Maxwell.py
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/conformalgeometryGAtest.py: ParseError:
bad input: type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Refactored ./doc/src/modules/galgebra/GA/coords.py
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/headerGAtest.py: ParseError: bad input:
type=0, value='', context=(u'\n', (26, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/hyperbolicGAtest.py: ParseError: bad
input: type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Can't parse
./doc/src/modules/galgebra/GA/reciprocalframeGAtest.py: ParseError:
bad input: type=5, value=u'        ', context=(u'', (1, 0))
RefactoringTool: Refactored ./doc/src/modules/galgebra/latex_ex/Maxwell.py
RefactoringTool: Refactored ./doc/src/modules/galgebra/latex_ex/latexdemo.py
RefactoringTool: Refactored ./examples/all.py
RefactoringTool: Refactored ./examples/advanced/curvilinear_coordinates.py
RefactoringTool: Refactored ./examples/advanced/fem.py
RefactoringTool: Refactored ./examples/advanced/gibbs_phenomenon.py
RefactoringTool: Refactored ./examples/advanced/pidigits.py
RefactoringTool: Refactored ./examples/advanced/plotting.py
RefactoringTool: Refactored ./examples/advanced/qft.py
RefactoringTool: Refactored ./examples/advanced/relativity.py
RefactoringTool: Refactored ./examples/beginner/basic.py
RefactoringTool: Refactored ./examples/beginner/differentiation.py
RefactoringTool: Refactored ./examples/beginner/expansion.py
RefactoringTool: Refactored ./examples/beginner/functions.py
RefactoringTool: Refactored ./examples/beginner/limits_examples.py
RefactoringTool: Refactored ./examples/beginner/precision.py
RefactoringTool: Refactored ./examples/beginner/print_pretty.py
RefactoringTool: Refactored ./examples/beginner/series.py
RefactoringTool: Refactored ./examples/beginner/substitution.py
RefactoringTool: Refactored ./examples/intermediate/differential_equations.py
RefactoringTool: No changes to ./examples/intermediate/mplot2d.py
RefactoringTool: Refactored ./examples/intermediate/trees.py
RefactoringTool: Refactored ./examples/intermediate/vandermonde.py
RefactoringTool: Refactored ./sympy/__init__.py
RefactoringTool: Refactored ./sympy/abc.py
RefactoringTool: Refactored ./sympy/benchmarks/bench_symbench.py
RefactoringTool: Refactored ./sympy/concrete/__init__.py
RefactoringTool: Refactored ./sympy/concrete/gosper.py
RefactoringTool: Refactored ./sympy/concrete/products.py
RefactoringTool: Refactored ./sympy/concrete/summations.py
RefactoringTool: Refactored ./sympy/concrete/sums_products.py
RefactoringTool: Refactored ./sympy/concrete/tests/test_sums_products.py
RefactoringTool: Refactored ./sympy/core/__init__.py
RefactoringTool: Refactored ./sympy/core/add.py
RefactoringTool: Refactored ./sympy/core/assumptions.py
RefactoringTool: Refactored ./sympy/core/ast_parser.py
RefactoringTool: Refactored ./sympy/core/ast_parser_python24.py
RefactoringTool: Refactored ./sympy/core/basic.py
RefactoringTool: Refactored ./sympy/core/cache.py
RefactoringTool: Refactored ./sympy/core/decorators.py
RefactoringTool: Refactored ./sympy/core/evalf.py
Traceback (most recent call last):
  File "/home/ondrej/repos/2to3/2to3", line 5, in <module>
    sys.exit(main("lib2to3.fixes"))
  File "/home/ondrej/repos/2to3/lib2to3/main.py", line 157, in main
    options.processes)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 545, in refactor
    items, write, doctests_only)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 207, in refactor
    self.refactor_dir(dir_or_file, write, doctests_only)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 225, in refactor_dir
    self.refactor_file(fullname, write, doctests_only)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 585, in refactor_file
    *args, **kwargs)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 260, in refactor_file
    tree = self.refactor_string(input, filename)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 286, in
refactor_string
    self.refactor_tree(tree, name)
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 320, in refactor_tree
    self.traverse_by(self.post_order_heads, tree.post_order())
  File "/home/ondrej/repos/2to3/lib2to3/refactor.py", line 346, in traverse_by
    str(new) != str(node)):
  File "/home/ondrej/repos/2to3/lib2to3/pytree.py", line 226, in __str__
    return unicode(self).encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03b1' in
position 1248: ordinal not in range(128)



am I invoking it in the correct way? I just downloaded the svn
repository into ~/repos/2to3/.

I cannot easily test bug 2), because even this fails with the
UnicodeEncodeError:

python2.6 ~/repos/2to3/2to3 -w .

>
> [1] http://svn.python.org/projects/sandbox/trunk/2to3
>
>>
>> Besides the 3 bugs above, there are lots of small annoyances, like
>> that sys.version_info has changed, so one has to use tuple() around
>> it, that .sort() doesn't accept a cmp() like callback anymore, and
>> similar things, which hopefully should be relatively easy to fix in a
>> manner that works in all pythons.
>
> What are you doing with sys.version_info which makes it incompatible?

I had to apply this patch:

         v = sys.version_info
-        python_version = "%s.%s.%s-%s-%s" % v
+        python_version = "%s.%s.%s-%s-%s" % tuple(v)


the "-" line works in python3.0 and below, but stops working in
python3.1. The fix is trivial, let's not loose time with it. After
fixing the 2to3 tool, I have a serious problem that the __hash__
method is defined in the Basic() class, but in it's subclasses (Number
and Symbol):

Number.__hash__

returns None, but

Symbol.__hash__

correctly points to the Basic.hash method. As a result, Number is not
hashable and this breaks sympy. I already spent 2 hours debugging it,
so far the only clue is that __slots__ influence it, but so far I
wasn't able to isolate a simple failing testcase (if I create a simple
class hierarchy, it works in python3.1), the problem is that it's not
easy to delete just part of sympy -- it's either all or nothing, and
thus it's very difficult to figure out why python3 behaves as it
behaves.

If anyone on this list has experience with debugging such kind of a
problem in python 3, I would be interested in any input.

Ondrej

From ondrej at certik.cz  Mon Jul  6 03:36:13 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 5 Jul 2009 19:36:13 -0600
Subject: [Python-porting] bugs in 2to3 tools when porting sympy
In-Reply-To: <85b5c3130907051754y39a60b29s80475b77955c0fc4@mail.gmail.com>
References: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>
	<1afaf6160907051721i5d85d44ew2399a40441e5b36b@mail.gmail.com>
	<85b5c3130907051754y39a60b29s80475b77955c0fc4@mail.gmail.com>
Message-ID: <85b5c3130907051836qa0f84ddsb18a40efa19a358c@mail.gmail.com>

On Sun, Jul 5, 2009 at 6:54 PM, Ondrej Certik<ondrej at certik.cz> wrote:
[...]
> I have a serious problem that the __hash__
> method is defined in the Basic() class, but in it's subclasses (Number
> and Symbol):
>
> Number.__hash__
>
> returns None, but
>
> Symbol.__hash__
>
> correctly points to the Basic.hash method. As a result, Number is not
> hashable and this breaks sympy. I already spent 2 hours debugging it,
> so far the only clue is that __slots__ influence it, but so far I
> wasn't able to isolate a simple failing testcase (if I create a simple
> class hierarchy, it works in python3.1), the problem is that it's not
> easy to delete just part of sympy -- it's either all or nothing, and
> thus it's very difficult to figure out why python3 behaves as it
> behaves.
>
> If anyone on this list has experience with debugging such kind of a
> problem in python 3, I would be interested in any input.

Here is how to reproduce it:

$ git clone git://github.com/certik/sympy.git
$ cd sympy
$ git checkout -b py3-hash-bug origin/py3-hash-bug
$ SYMPY_USE_CACHE=no python3.1 t.py
1
2
Atom: <function __hash__ at 0x1c0c270>
Number: None
Integer: None
Symbol: <function __hash__ at 0x1c0c270>

as you can see, the __hash__ method is defined in Basic and all Atom,
Number, Integer and Symbol are subclasses of Basic.

I must say I am totally puzzled as how this is possible.

Ondrej

P.S. The "SYMPY_USE_CACHE=no" is necessary, because otherwise sympy
will not import (due to this hash bug), as it can't cash things.

From amauryfa at gmail.com  Mon Jul  6 10:01:19 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 6 Jul 2009 10:01:19 +0200
Subject: [Python-porting] bugs in 2to3 tools when porting sympy
In-Reply-To: <85b5c3130907051836qa0f84ddsb18a40efa19a358c@mail.gmail.com>
References: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>
	<1afaf6160907051721i5d85d44ew2399a40441e5b36b@mail.gmail.com>
	<85b5c3130907051754y39a60b29s80475b77955c0fc4@mail.gmail.com>
	<85b5c3130907051836qa0f84ddsb18a40efa19a358c@mail.gmail.com>
Message-ID: <e27efe130907060101m4cc67f7ft33c2c05839db077c@mail.gmail.com>

Hello,

2009/7/6 Ondrej Certik <ondrej at certik.cz>:
> On Sun, Jul 5, 2009 at 6:54 PM, Ondrej Certik<ondrej at certik.cz> wrote:
> [...]
>> I have a serious problem that the __hash__
>> method is defined in the Basic() class, but in it's subclasses (Number
>> and Symbol):
>>
>> Number.__hash__
>>
>> returns None, but
>>
>> Symbol.__hash__
>>
>> correctly points to the Basic.hash method.

Number certainly defines a __eq__ method, which blocks the inheritance
of __hash__.
See the last paragraph of
http://docs.python.org/dev/py3k/reference/datamodel.html#object.__hash__

-- 
Amaury Forgeot d'Arc

From ondrej at certik.cz  Mon Jul  6 11:14:40 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 6 Jul 2009 03:14:40 -0600
Subject: [Python-porting] bugs in 2to3 tools when porting sympy
In-Reply-To: <e27efe130907060101m4cc67f7ft33c2c05839db077c@mail.gmail.com>
References: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>
	<1afaf6160907051721i5d85d44ew2399a40441e5b36b@mail.gmail.com>
	<85b5c3130907051754y39a60b29s80475b77955c0fc4@mail.gmail.com>
	<85b5c3130907051836qa0f84ddsb18a40efa19a358c@mail.gmail.com>
	<e27efe130907060101m4cc67f7ft33c2c05839db077c@mail.gmail.com>
Message-ID: <85b5c3130907060214i43bda26bia60793105a94a074@mail.gmail.com>

On Mon, Jul 6, 2009 at 2:01 AM, Amaury Forgeot d'Arc<amauryfa at gmail.com> wrote:
> Hello,
>
> 2009/7/6 Ondrej Certik <ondrej at certik.cz>:
>> On Sun, Jul 5, 2009 at 6:54 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>> [...]
>>> I have a serious problem that the __hash__
>>> method is defined in the Basic() class, but in it's subclasses (Number
>>> and Symbol):
>>>
>>> Number.__hash__
>>>
>>> returns None, but
>>>
>>> Symbol.__hash__
>>>
>>> correctly points to the Basic.hash method.
>
> Number certainly defines a __eq__ method, which blocks the inheritance
> of __hash__.
> See the last paragraph of
> http://docs.python.org/dev/py3k/reference/datamodel.html#object.__hash__

That was it! Many thanks for the hint, without it I would spend a lot
more hours on this.
So Python3.0 seems to be more strict than python2.6, since that code
is in sympy for a long time and it always worked in python2.4 up to
2.6. The fix is trivial, I just assign the __hash__ method from a
parent and things start working.

There are other unrelated failures, but those I will tackle myself.

Thanks again for the help,
Ondrej

From benjamin at python.org  Sat Jul 11 23:45:55 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 11 Jul 2009 16:45:55 -0500
Subject: [Python-porting] bugs in 2to3 tools when porting sympy
In-Reply-To: <85b5c3130907051754y39a60b29s80475b77955c0fc4@mail.gmail.com>
References: <85b5c3130907051522q580adb4id1fd7e3bf4ff5210@mail.gmail.com>
	<1afaf6160907051721i5d85d44ew2399a40441e5b36b@mail.gmail.com>
	<85b5c3130907051754y39a60b29s80475b77955c0fc4@mail.gmail.com>
Message-ID: <1afaf6160907111445h77bc4374mfe1c63ab2010a141@mail.gmail.com>

2009/7/5 Ondrej Certik <ondrej at certik.cz>:
> On Sun, Jul 5, 2009 at 6:21 PM, Benjamin Peterson<benjamin at python.org> wrote:
>> Could you try with the latest version of 2to3 from the sandbox? [1]
>> I've recently fixed I think all of the bugs you've described.
>
> I tested the bug 1):

Ok. Thank you. You discovered several more bugs. Please try again with
my latest fixes.


-- 
Regards,
Benjamin