From barry at python.org  Wed May 16 20:57:02 2012
From: barry at python.org (Barry Warsaw)
Date: Wed, 16 May 2012 14:57:02 -0400
Subject: [Python-porting] Python 2/3 compatible distutils installation?
Message-ID: <20120516145702.1eb0b12e@resist.wooz.org>

Here's an interesting porting issue that came up today.  I'm not crossposting
this to distutils-sig, but I will if no one here has come up with a solution.

I'm looking at testtools[1], which Jerry Seutter has done a good, initial port
of to Python 3.  His merge proposal, along with the diff is here[2].  My
branch which fixes a few of Robert Collins' comments is here[3].

One of the tricky problems is with re-raise syntax, which can't be
compatibly represented syntactically across Python 2 and 3.  The six module
has a hack for this, but testtools uses its own trick, which is to provide a
_compat2x.py and a _compat3x.py file containing the proper syntax.

This works well at execution time, because a simple sys.version_info check can
be used to decide which module gets imported; the other is for all intents and
purposes, ignored.

The problem comes when installing the package with distutils.  The
byte-compilation of _compat2x.py throws a SyntaxError, of course rightly so.

The question I have is how can we avoid trying to byte compile _compat2x.py
when the package is installed via Python 3?  I've tried various distutils
command overrides and hacks (including trying to load a MANIFEST3.in which
exclusions that file), but haven't hit upon the right magic to make it work in
all cases.

Has anybody encountered a similar situation, and if so, how have you worked
around this?  Disgusting kludges welcome. :)

Cheers,
-Barry

[1] http://launchpad.net/testtools
[2] https://code.launchpad.net/~jseutter/testtools/py3_again/+merge/105569
[3] https://code.launchpad.net/~barry/testtools/py3
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120516/47f5ca20/attachment.pgp>

From chris.jerdonek at gmail.com  Wed May 16 21:24:29 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 16 May 2012 12:24:29 -0700
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <20120516145702.1eb0b12e@resist.wooz.org>
References: <20120516145702.1eb0b12e@resist.wooz.org>
Message-ID: <CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>

On Wed, May 16, 2012 at 11:57 AM, Barry Warsaw <barry at python.org> wrote:
> The question I have is how can we avoid trying to byte compile _compat2x.py
> when the package is installed via Python 3? ?I've tried various distutils
> command overrides and hacks (including trying to load a MANIFEST3.in which
> exclusions that file), but haven't hit upon the right magic to make it work in
> all cases.

I also encountered this issue recently.  Can you just avoid importing
anything from your package from within setup.py?  It looks like you're
importing your package mostly just to get the __version__ number in
your package's __init__.py (though I also see you use it in the
cmdclass argument to setup(), which might require a different
work-around).

The consequence for me was that I needed to maintain the version
string for my project in two places: both setup.py and my package's
__init__.py (and I added a unit test to check that they were the
same).  I rationalized this by telling myself that the script
responsible for installing a project shouldn't need to be able to run
the project: it should be independent.

--Chris

From barry at python.org  Wed May 16 23:36:25 2012
From: barry at python.org (Barry Warsaw)
Date: Wed, 16 May 2012 17:36:25 -0400
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
Message-ID: <20120516173625.42b9f22f@resist.wooz.org>

On May 16, 2012, at 12:24 PM, Chris Jerdonek wrote:

>I also encountered this issue recently.  Can you just avoid importing
>anything from your package from within setup.py?

It's not actually *my* setup.py, but I get what you're saying.  In my own
packages, I do try to avoid those types of imports.

>It looks like you're importing your package mostly just to get the
>__version__ number in your package's __init__.py (though I also see you use
>it in the cmdclass argument to setup(), which might require a different
>work-around).

Yep.  It seems like the testtools.TestCommand is required.

>The consequence for me was that I needed to maintain the version
>string for my project in two places: both setup.py and my package's
>__init__.py (and I added a unit test to check that they were the
>same).  I rationalized this by telling myself that the script
>responsible for installing a project shouldn't need to be able to run
>the project: it should be independent.

In my own packages, I still keep the version string in my __init__.py, but I
don't import it from setup.py to get at it.  Instead, I usually just open the
file and grep its lines for the value.  I also have a helper to make this
easier, e.g.

http://bazaar.launchpad.net/~barry/flufl.enum/trunk/view/head:/setup_helpers.py

and its use in my setup.py

http://bazaar.launchpad.net/~barry/flufl.enum/trunk/view/head:/setup.py

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120516/561d4773/attachment.pgp>

From chris.jerdonek at gmail.com  Thu May 17 00:16:10 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 16 May 2012 15:16:10 -0700
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <20120516173625.42b9f22f@resist.wooz.org>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
	<20120516173625.42b9f22f@resist.wooz.org>
Message-ID: <CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>

On Wed, May 16, 2012 at 2:36 PM, Barry Warsaw <barry at python.org> wrote:
> On May 16, 2012, at 12:24 PM, Chris Jerdonek wrote:
>
>>I also encountered this issue recently. ?Can you just avoid importing
>>anything from your package from within setup.py?
>
> It's not actually *my* setup.py, but I get what you're saying. ?In my own
> packages, I do try to avoid those types of imports.

Sorry, I meant to say the package that you're helping with.

I realize now that the issue I had might be completely different,
because I was using 2to3 and testtools protects the import with a
version check, etc.  So perhaps my suggestion won't help.  What causes
the install process to byte-compile the files?

Also, thanks for the idea and pointers re: grabbing the version
number.  That idea had crossed my mind, too, but I didn't implement
it.

--Chris

From barry at python.org  Thu May 17 01:34:11 2012
From: barry at python.org (Barry Warsaw)
Date: Wed, 16 May 2012 19:34:11 -0400
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
	<20120516173625.42b9f22f@resist.wooz.org>
	<CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>
Message-ID: <20120516193411.478a12df@resist.wooz.org>

On May 16, 2012, at 03:16 PM, Chris Jerdonek wrote:

>What causes the install process to byte-compile the files?

That's the question. :)  By which I mean, the files get byte compiled on
installation, and I've tried various things to have it ignore _compat2x.py
when installing in Python 3, but with no luck.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120516/7be57f39/attachment-0001.pgp>

From merwok at netwok.org  Thu May 17 02:13:30 2012
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Wed, 16 May 2012 20:13:30 -0400
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <20120516193411.478a12df@resist.wooz.org>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
	<20120516173625.42b9f22f@resist.wooz.org>
	<CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>
	<20120516193411.478a12df@resist.wooz.org>
Message-ID: <4FB442AA.2020706@netwok.org>

Le 16/05/2012 19:34, Barry Warsaw a ?crit :
> On May 16, 2012, at 03:16 PM, Chris Jerdonek wrote:
>> What causes the install process to byte-compile the files?
> That's the question. :)  By which I mean, the files get byte compiled on
> installation, and I've tried various things to have it ignore _compat2x.py
> when installing in Python 3, but with no luck.

distutils does that.  As far as I know you can either disable
byte-compilation wholesale, or have it for all modules.  I don?t think
even a dirty hack like registering the _compat?.py files as package_data
would work, because the byte-compilation function is applied to the
whole build or install dir (but try it anyway, we never know).

See http://bugs.python.org/issue10530 for a distutils2 feature request
about this same use case.

Cheers

From chris.jerdonek at gmail.com  Thu May 17 03:38:23 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 16 May 2012 18:38:23 -0700
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <20120516193411.478a12df@resist.wooz.org>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
	<20120516173625.42b9f22f@resist.wooz.org>
	<CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>
	<20120516193411.478a12df@resist.wooz.org>
Message-ID: <CAOTb1wf0Sri_pY9YnLQfOV87HX84ShTgb6AzNRtnR98D3H2m6w@mail.gmail.com>

On Wed, May 16, 2012 at 4:34 PM, Barry Warsaw <barry at python.org> wrote:
> On May 16, 2012, at 03:16 PM, Chris Jerdonek wrote:
>
> That's the question. :) ?By which I mean, the files get byte compiled on
> installation, and I've tried various things to have it ignore _compat2x.py
> when installing in Python 3, but with no luck.

Have you thought about storing the two files with an extension other
than .py?  You could argue that from Python 3's perspective, for
example, the compat2 file isn't really a Python file.  You could write
your setup script so that it copies the correct file into place at
compat.py before calling setup(), and then cleans up afterwards.  This
way, the distutils code could proceed normally.  Also, you wouldn't
need special version-checking code in the actual package because then
both versions would be importing compat when installed.

--Chris

From barry at python.org  Thu May 17 04:10:41 2012
From: barry at python.org (Barry Warsaw)
Date: Wed, 16 May 2012 22:10:41 -0400
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <4FB442AA.2020706@netwok.org>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
	<20120516173625.42b9f22f@resist.wooz.org>
	<CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>
	<20120516193411.478a12df@resist.wooz.org>
	<4FB442AA.2020706@netwok.org>
Message-ID: <20120516221041.3bf2bc70@resist.wooz.org>

On May 16, 2012, at 08:13 PM, ?ric Araujo wrote:

>See http://bugs.python.org/issue10530 for a distutils2 feature request
>about this same use case.

Thanks, nosied.

-Barry

From barry at python.org  Thu May 17 04:12:41 2012
From: barry at python.org (Barry Warsaw)
Date: Wed, 16 May 2012 22:12:41 -0400
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <CAOTb1wf0Sri_pY9YnLQfOV87HX84ShTgb6AzNRtnR98D3H2m6w@mail.gmail.com>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
	<20120516173625.42b9f22f@resist.wooz.org>
	<CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>
	<20120516193411.478a12df@resist.wooz.org>
	<CAOTb1wf0Sri_pY9YnLQfOV87HX84ShTgb6AzNRtnR98D3H2m6w@mail.gmail.com>
Message-ID: <20120516221241.0fda10fc@resist.wooz.org>

On May 16, 2012, at 06:38 PM, Chris Jerdonek wrote:

>Have you thought about storing the two files with an extension other
>than .py?  You could argue that from Python 3's perspective, for
>example, the compat2 file isn't really a Python file.  You could write
>your setup script so that it copies the correct file into place at
>compat.py before calling setup(), and then cleans up afterwards.  This
>way, the distutils code could proceed normally.  Also, you wouldn't
>need special version-checking code in the actual package because then
>both versions would be importing compat when installed.

It's an interesting idea, thanks.  I'll point upstream at this thread and see
if they want to explore it more or if they want to go in the direction of
using six (which would eliminate the need to use the _compatX.py files in this
particular case).

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120516/cbf32020/attachment.pgp>

From brett at python.org  Thu May 17 17:14:05 2012
From: brett at python.org (Brett Cannon)
Date: Thu, 17 May 2012 11:14:05 -0400
Subject: [Python-porting] Python 2/3 compatible distutils installation?
In-Reply-To: <20120516221241.0fda10fc@resist.wooz.org>
References: <20120516145702.1eb0b12e@resist.wooz.org>
	<CAOTb1wcAynAZNJSV92mjTEvi=J4+XOb8=NEK5RsfQ7k24rxRdw@mail.gmail.com>
	<20120516173625.42b9f22f@resist.wooz.org>
	<CAOTb1we8CaMwZM+9p77pdwZ_pVKMKdO-zD8BewdiAyV7gpwD-w@mail.gmail.com>
	<20120516193411.478a12df@resist.wooz.org>
	<CAOTb1wf0Sri_pY9YnLQfOV87HX84ShTgb6AzNRtnR98D3H2m6w@mail.gmail.com>
	<20120516221241.0fda10fc@resist.wooz.org>
Message-ID: <CAP1=2W6GUTWjwNK9YfyLPOKo58jrLjUpqcx=aiejyA8b_tTWSg@mail.gmail.com>

On Wed, May 16, 2012 at 10:12 PM, Barry Warsaw <barry at python.org> wrote:

> On May 16, 2012, at 06:38 PM, Chris Jerdonek wrote:
>
> >Have you thought about storing the two files with an extension other
> >than .py?  You could argue that from Python 3's perspective, for
> >example, the compat2 file isn't really a Python file.  You could write
> >your setup script so that it copies the correct file into place at
> >compat.py before calling setup(), and then cleans up afterwards.  This
> >way, the distutils code could proceed normally.  Also, you wouldn't
> >need special version-checking code in the actual package because then
> >both versions would be importing compat when installed.
>
> It's an interesting idea, thanks.  I'll point upstream at this thread and
> see
> if they want to explore it more or if they want to go in the direction of
> using six (which would eliminate the need to use the _compatX.py files in
> this
> particular case).


Your other option is to keep the code in string literals and then do the
proper compile/exec step just like import does. Basically do you want the
hack in distutils or the modules themselves.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120517/2f7f8439/attachment.html>

From barry at python.org  Thu May 24 18:10:16 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 24 May 2012 12:10:16 -0400
Subject: [Python-porting] oauth for Python 3
Message-ID: <20120524121016.4934e008@resist.wooz.org>

One of the packages that I'm constantly bumping up against as I port more code
to Python 3 is oauth.  We really need a semi-official (or at least popular
<wink>) Python 3 compatible oauth client library.

I'm CC'ing Jason DeRose since I've had several conversations with him about
this, and I believe he has a package that would fit the bill, although I'm not
sure if it's available on PyPI yet.

I know I've spoken to at least one other person who knew of, or was working
on, a Python 3 compatible oauth library, but I can't remember who that is.

The most popular oauth package on the Cheeseshop is oauth 1.0.1, but afaik
upstream has abandoned it, and it has had a new version on PyPI in since
September 2009.  I'm not personally very keen on adopting it in order to port
it.

There are many other hits for 'oauth' on PyPI, but none of the client
libraries are Python 3 compatible already afaict.

It would be really fantastic if a Python 3 compatible oauth library existed on
PyPI, and it would be even better (maybe) if it was API compatible with oauth
1.0.1.

Is anybody else out there trying to do oauth client interaction in Python 3,
and if so, what are you using?  Jason, maybe you can speak about your library
and whether it can or will be available on PyPI?

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120524/440fd08e/attachment.pgp>

From brian at python.org  Thu May 24 18:17:36 2012
From: brian at python.org (Brian Curtin)
Date: Thu, 24 May 2012 11:17:36 -0500
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <20120524121016.4934e008@resist.wooz.org>
References: <20120524121016.4934e008@resist.wooz.org>
Message-ID: <CAD+XWwq7LLBQCZN050MVBRd+M7H-Xd5E_+8ihLZDZgpvzC9_Jg@mail.gmail.com>

On Thu, May 24, 2012 at 11:10 AM, Barry Warsaw <barry at python.org> wrote:
> One of the packages that I'm constantly bumping up against as I port more code
> to Python 3 is oauth. ?We really need a semi-official (or at least popular
> <wink>) Python 3 compatible oauth client library.
>
> I'm CC'ing Jason DeRose since I've had several conversations with him about
> this, and I believe he has a package that would fit the bill, although I'm not
> sure if it's available on PyPI yet.
>
> I know I've spoken to at least one other person who knew of, or was working
> on, a Python 3 compatible oauth library, but I can't remember who that is.

Perhaps it was Rodney Dawes (also at Canonical).
https://launchpad.net/poauth is on his radar for moving U1 towards 3.x
support.

(I saw you ask the same thing on Twitter but I figured I'd answer here
as well for the benefit of the group)

From guido at python.org  Thu May 24 18:27:51 2012
From: guido at python.org (Guido van Rossum)
Date: Thu, 24 May 2012 09:27:51 -0700
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <CAD+XWwq7LLBQCZN050MVBRd+M7H-Xd5E_+8ihLZDZgpvzC9_Jg@mail.gmail.com>
References: <20120524121016.4934e008@resist.wooz.org>
	<CAD+XWwq7LLBQCZN050MVBRd+M7H-Xd5E_+8ihLZDZgpvzC9_Jg@mail.gmail.com>
Message-ID: <CAP7+vJKPEoXL-d7q3rpP36Ewr9Abxz6XWd2=qtEBm3dFYk0-Fw@mail.gmail.com>

The OAuth protocol is already being abandoned in favor of OAuth2.
Whatever package you choose please be sure it uses OAuth2. Google has
an "oauth2client" library but sadly it is horribly over-engineered and
I cannot recommend it.

On Thu, May 24, 2012 at 9:17 AM, Brian Curtin <brian at python.org> wrote:
> On Thu, May 24, 2012 at 11:10 AM, Barry Warsaw <barry at python.org> wrote:
>> One of the packages that I'm constantly bumping up against as I port more code
>> to Python 3 is oauth. ?We really need a semi-official (or at least popular
>> <wink>) Python 3 compatible oauth client library.
>>
>> I'm CC'ing Jason DeRose since I've had several conversations with him about
>> this, and I believe he has a package that would fit the bill, although I'm not
>> sure if it's available on PyPI yet.
>>
>> I know I've spoken to at least one other person who knew of, or was working
>> on, a Python 3 compatible oauth library, but I can't remember who that is.
>
> Perhaps it was Rodney Dawes (also at Canonical).
> https://launchpad.net/poauth is on his radar for moving U1 towards 3.x
> support.
>
> (I saw you ask the same thing on Twitter but I figured I'd answer here
> as well for the benefit of the group)
> _______________________________________________
> Python-porting mailing list
> Python-porting at python.org
> http://mail.python.org/mailman/listinfo/python-porting



-- 
--Guido van Rossum (python.org/~guido)

From barry at python.org  Thu May 24 22:16:46 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 24 May 2012 16:16:46 -0400
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <CAP7+vJKPEoXL-d7q3rpP36Ewr9Abxz6XWd2=qtEBm3dFYk0-Fw@mail.gmail.com>
References: <20120524121016.4934e008@resist.wooz.org>
	<CAD+XWwq7LLBQCZN050MVBRd+M7H-Xd5E_+8ihLZDZgpvzC9_Jg@mail.gmail.com>
	<CAP7+vJKPEoXL-d7q3rpP36Ewr9Abxz6XWd2=qtEBm3dFYk0-Fw@mail.gmail.com>
Message-ID: <20120524161646.13bf8838@resist.wooz.org>

On May 24, 2012, at 09:27 AM, Guido van Rossum wrote:

>The OAuth protocol is already being abandoned in favor of OAuth2.
>Whatever package you choose please be sure it uses OAuth2. Google has
>an "oauth2client" library but sadly it is horribly over-engineered and
>I cannot recommend it.

Currently, Launchpad doesn't support OAuth2, so while I agree this would be an
important feature for a general purpose Python 3 compatible library, it's not
a requirement for my immediate needs.

Rodney Dawes is the other person who was talking about a Python 3 OAuth
library, but he is also only needs OAuth1 atm, and his library is not yet
ready for a Python 3 compatible release.  So I think right now, there is
basically nothing available.

I'm going to experiment with porting the original oauth library to Python 3.
If it works out, I'll figure out what to do from there.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120524/e323d9b4/attachment.pgp>

From barry at python.org  Thu May 24 22:20:03 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 24 May 2012 16:20:03 -0400
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <CACZ=4fuH6bULY4L1etEgvF4EVL5=YKYuHMs0h=G1XZ5dJpnD2Q@mail.gmail.com>
References: <20120524121016.4934e008@resist.wooz.org>
	<CACZ=4fuH6bULY4L1etEgvF4EVL5=YKYuHMs0h=G1XZ5dJpnD2Q@mail.gmail.com>
Message-ID: <20120524162003.14863c79@resist.wooz.org>

On May 24, 2012, at 10:51 AM, Jason Gerard DeRose wrote:

>I don't yet have a Python3 OAuth library, but I implemented OAuth 1a
>internally in my Python3 CouchDB "REST adapter", which I still need to get
>up on PyPI, but can also be found on Launchpad:
>
>https://launchpad.net/microfiber
>
>I only implemented the client functionality, but as far as I know, that's all
>that's needed for libraries like launchpadlib. I felt the python-oauth API
>was way over complicated, so I wrote a simple, functional implementation:
>
>http://bazaar.launchpad.net/~microfiber/microfiber/trunk/view/head:/microfiber.py#L152
>
>My plan is to split this out into its own module, and then to add a
>python-oauth compatibility API (at least for the client functionality
>actually being used). And then I'll try porting lauchpadlib to this
>(sticking with Python2 at first).
>
>Realistically, I wont have a chance to do this till I return from Linaro
>Connect on May 2nd.
>
>Python3 FTW!

Thanks for the update.  See my previous follow up.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120524/396a1443/attachment.pgp>

From roland.hedberg at adm.umu.se  Thu May 24 23:15:22 2012
From: roland.hedberg at adm.umu.se (Roland Hedberg)
Date: Thu, 24 May 2012 23:15:22 +0200
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <20120524162003.14863c79@resist.wooz.org>
References: <20120524121016.4934e008@resist.wooz.org>
	<CACZ=4fuH6bULY4L1etEgvF4EVL5=YKYuHMs0h=G1XZ5dJpnD2Q@mail.gmail.com>
	<20120524162003.14863c79@resist.wooz.org>
Message-ID: <1AF29645-CF97-4A0D-84E9-12F0A9D85E66@adm.umu.se>


24 maj 2012 kl. 22:20 skrev Barry Warsaw:

> On May 24, 2012, at 10:51 AM, Jason Gerard DeRose wrote:
> 
>> I don't yet have a Python3 OAuth library, but I implemented OAuth 1a
>> internally in my Python3 CouchDB "REST adapter", which I still need to get
>> up on PyPI, but can also be found on Launchpad:

I've written a OpenID Connect library in Python 2.7 .
It should be rather straight forward to translate it to Python3.
Since OpenID Connect is an extension of OAuth2, it contains a OAuth2 library too.

https://github.com/rohe/pyoidc

The OpenID Connect implementation is done on behalf of Kantara and is done in close cooperation with the team that writes the specification.
It's the basis of a functional test tool, already in use to verify other implementations.
So it is in essence the reference implementation.

-- Roland
------------------------------------------------------
Roland Hedberg
IT Architect/Senior Researcher
ICT Services and System Development (ITS) 
Ume? University 
SE-901 87 Ume?, Sweden	
Phone +46 90 786 68 44
Mobile +46 70 696 68 44 
www.its.umu.se 


From guido at python.org  Thu May 24 23:55:28 2012
From: guido at python.org (Guido van Rossum)
Date: Thu, 24 May 2012 14:55:28 -0700
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <20120524161646.13bf8838@resist.wooz.org>
References: <20120524121016.4934e008@resist.wooz.org>
	<CAD+XWwq7LLBQCZN050MVBRd+M7H-Xd5E_+8ihLZDZgpvzC9_Jg@mail.gmail.com>
	<CAP7+vJKPEoXL-d7q3rpP36Ewr9Abxz6XWd2=qtEBm3dFYk0-Fw@mail.gmail.com>
	<20120524161646.13bf8838@resist.wooz.org>
Message-ID: <CAP7+vJ+RVNFuCPrKbEpeL3JxLQkw7Ge+bRELHCNXPnTpS2ha+w@mail.gmail.com>

On Thu, May 24, 2012 at 1:16 PM, Barry Warsaw <barry at python.org> wrote:
> On May 24, 2012, at 09:27 AM, Guido van Rossum wrote:
>
>>The OAuth protocol is already being abandoned in favor of OAuth2.
>>Whatever package you choose please be sure it uses OAuth2. Google has
>>an "oauth2client" library but sadly it is horribly over-engineered and
>>I cannot recommend it.
>
> Currently, Launchpad doesn't support OAuth2, so while I agree this would be an
> important feature for a general purpose Python 3 compatible library, it's not
> a requirement for my immediate needs.

Yeah, but your immediate needs don't necessarily fit the standard library. :-)

> Rodney Dawes is the other person who was talking about a Python 3 OAuth
> library, but he is also only needs OAuth1 atm, and his library is not yet
> ready for a Python 3 compatible release. ?So I think right now, there is
> basically nothing available.
>
> I'm going to experiment with porting the original oauth library to Python 3.
> If it works out, I'll figure out what to do from there.

I would object to including something in the stdlib that's already
known to be out of date.

-- 
--Guido van Rossum (python.org/~guido)

From barry at python.org  Fri May 25 00:11:31 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 24 May 2012 18:11:31 -0400
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <CAP7+vJ+RVNFuCPrKbEpeL3JxLQkw7Ge+bRELHCNXPnTpS2ha+w@mail.gmail.com>
References: <20120524121016.4934e008@resist.wooz.org>
	<CAD+XWwq7LLBQCZN050MVBRd+M7H-Xd5E_+8ihLZDZgpvzC9_Jg@mail.gmail.com>
	<CAP7+vJKPEoXL-d7q3rpP36Ewr9Abxz6XWd2=qtEBm3dFYk0-Fw@mail.gmail.com>
	<20120524161646.13bf8838@resist.wooz.org>
	<CAP7+vJ+RVNFuCPrKbEpeL3JxLQkw7Ge+bRELHCNXPnTpS2ha+w@mail.gmail.com>
Message-ID: <20120524181131.734c5c82@resist.wooz.org>

On May 24, 2012, at 02:55 PM, Guido van Rossum wrote:

>> Currently, Launchpad doesn't support OAuth2, so while I agree this would be
>> an important feature for a general purpose Python 3 compatible library,
>> it's not a requirement for my immediate needs.
>
>Yeah, but your immediate needs don't necessarily fit the standard library. :-)
>> I'm going to experiment with porting the original oauth library to Python 3.
>> If it works out, I'll figure out what to do from there.
>
>I would object to including something in the stdlib that's already
>known to be out of date.

Oh, I completely agree about that.  I wasn't thinking about the stdlib, but
eventually when we have a modern, standards-compliant, battle-tested third
party library on the Cheeseshop, I do think it would make a nice addition to
Python 3, but probably not until 3.3.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120524/6f29cb69/attachment.pgp>

From barry at python.org  Fri May 25 00:16:04 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 24 May 2012 18:16:04 -0400
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <1AF29645-CF97-4A0D-84E9-12F0A9D85E66@adm.umu.se>
References: <20120524121016.4934e008@resist.wooz.org>
	<CACZ=4fuH6bULY4L1etEgvF4EVL5=YKYuHMs0h=G1XZ5dJpnD2Q@mail.gmail.com>
	<20120524162003.14863c79@resist.wooz.org>
	<1AF29645-CF97-4A0D-84E9-12F0A9D85E66@adm.umu.se>
Message-ID: <20120524181604.17165fb4@resist.wooz.org>

On May 24, 2012, at 11:15 PM, Roland Hedberg wrote:

>I've written a OpenID Connect library in Python 2.7 .  It should be rather
>straight forward to translate it to Python3.  Since OpenID Connect is an
>extension of OAuth2, it contains a OAuth2 library too.

I took a quick look at the install_requires from your setup.py.  It looks like
only M2Crypto still needs a Python 3 port.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120524/30d9401b/attachment.pgp>

From jderose at novacut.com  Thu May 24 18:51:50 2012
From: jderose at novacut.com (Jason Gerard DeRose)
Date: Thu, 24 May 2012 10:51:50 -0600
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <20120524121016.4934e008@resist.wooz.org>
References: <20120524121016.4934e008@resist.wooz.org>
Message-ID: <CACZ=4fuH6bULY4L1etEgvF4EVL5=YKYuHMs0h=G1XZ5dJpnD2Q@mail.gmail.com>

On Thu, May 24, 2012 at 10:10 AM, Barry Warsaw <barry at python.org> wrote:

> One of the packages that I'm constantly bumping up against as I port more
> code
> to Python 3 is oauth.  We really need a semi-official (or at least popular
> <wink>) Python 3 compatible oauth client library.
>
> I'm CC'ing Jason DeRose since I've had several conversations with him about
> this, and I believe he has a package that would fit the bill, although I'm
> not
> sure if it's available on PyPI yet.
>

I don't yet have a Python3 OAuth library, but I implemented OAuth 1a
internally in my Python3 CouchDB "REST adapter", which I still need to get
up on PyPI, but can also be found on Launchpad:

https://launchpad.net/microfiber

I only implemented the client functionality, but as far as I know, that's
all that's needed for libraries like launchpadlib. I felt the python-oauth
API was way over complicated, so I wrote a simple, functional
implementation:

http://bazaar.launchpad.net/~microfiber/microfiber/trunk/view/head:/microfiber.py#L152

My plan is to split this out into its own module, and then to add a
python-oauth compatibility API (at least for the client functionality
actually being used). And then I'll try porting lauchpadlib to this
(sticking with Python2 at first).

Realistically, I wont have a chance to do this till I return from Linaro
Connect on May 2nd.

Python3 FTW!


> I know I've spoken to at least one other person who knew of, or was working
> on, a Python 3 compatible oauth library, but I can't remember who that is.
>
> The most popular oauth package on the Cheeseshop is oauth 1.0.1, but afaik
> upstream has abandoned it, and it has had a new version on PyPI in since
> September 2009.  I'm not personally very keen on adopting it in order to
> port
> it.
>
> There are many other hits for 'oauth' on PyPI, but none of the client
> libraries are Python 3 compatible already afaict.
>
> It would be really fantastic if a Python 3 compatible oauth library
> existed on
> PyPI, and it would be even better (maybe) if it was API compatible with
> oauth
> 1.0.1.
>
> Is anybody else out there trying to do oauth client interaction in Python
> 3,
> and if so, what are you using?  Jason, maybe you can speak about your
> library
> and whether it can or will be available on PyPI?
>
> Cheers,
> -Barry
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20120524/a452a472/attachment.html>

From roland.hedberg at adm.umu.se  Sat May 26 15:35:32 2012
From: roland.hedberg at adm.umu.se (Roland Hedberg)
Date: Sat, 26 May 2012 15:35:32 +0200
Subject: [Python-porting] oauth for Python 3
In-Reply-To: <20120524181604.17165fb4@resist.wooz.org>
References: <20120524121016.4934e008@resist.wooz.org>
	<CACZ=4fuH6bULY4L1etEgvF4EVL5=YKYuHMs0h=G1XZ5dJpnD2Q@mail.gmail.com>
	<20120524162003.14863c79@resist.wooz.org>
	<1AF29645-CF97-4A0D-84E9-12F0A9D85E66@adm.umu.se>
	<20120524181604.17165fb4@resist.wooz.org>
Message-ID: <D24E8CC2-545A-4F5C-8EDB-F5EDC1F8CCAE@adm.umu.se>


25 maj 2012 kl. 00:16 skrev Barry Warsaw:

> On May 24, 2012, at 11:15 PM, Roland Hedberg wrote:
> 
>> I've written a OpenID Connect library in Python 2.7 .  It should be rather
>> straight forward to translate it to Python3.  Since OpenID Connect is an
>> extension of OAuth2, it contains a OAuth2 library too.
> 
> I took a quick look at the install_requires from your setup.py.  It looks like
> only M2Crypto still needs a Python 3 port.


Note though that this library was designed to be able to cope with continuous changing specifications.
So it's not optimized for speed or memory usage. 
It's also not battle tested as someone put it.
But it's a starting point.

-- Roland
------------------------------------------------------
Roland Hedberg
IT Architect/Senior Researcher
ICT Services and System Development (ITS) 
Ume? University 
SE-901 87 Ume?, Sweden	
Phone +46 90 786 68 44
Mobile +46 70 696 68 44 
www.its.umu.se 


From asmeurer at gmail.com  Sun May 27 06:58:13 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Sat, 26 May 2012 22:58:13 -0600
Subject: [Python-porting] Control of hash randomization
Message-ID: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>

Hi.

First, I want to apologize if this is the wrong list for this
question. If it is, kindly direct me to the correct one and I'll be on
my way.

I am attempting to "port" SymPy to Python 3.3.  We already have full
support for Python 3.2, but the new hash randomization in Python 3.3
has brought up a bunch of new failures in our test suite.  This is not
a surprise, as a lot of operations in SymPy are dependent on hash
values.  But expecting the problem does not entirely help use to solve
it.

My question is this: is there a way to get the random seed used for
hash randomization, and to print it out, and then to later input that
seed to the interpreter?  Without this ability, it becomes extremely
difficult to reproduce test failures.  The best bet is to restart the
interpreter multiple times until it occurs.  We already have
experienced in SymPy several odd bugs that were caused by very rare
seeds in the random module.  This seed is easy to set, though, and we
print it with each test run, so the handful of issues that have
cropped up we've been able to fix.  I'd like to have that same ability
with hash randomization.

I figured that this is a new feature, and people on this list are
supposed to be experts on new Python features, so my hope is that
either someone here will know the answer, or can point me in the right
direction to one.

If it is the case, as I fear/expect, that this is not possible, should
I open an issue in the Python bug tracker?  Should I raise the issue
on python-dev?  Has it been discussed before?  I realize that there
may be a security risk (imho a low one, though) in such a feature, so
it might not be immediately accepted.

Aaron Meurer

From asmeurer at gmail.com  Sun May 27 07:26:53 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Sat, 26 May 2012 23:26:53 -0600
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
Message-ID: <CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>

I found half of my answer.  You can set it with the PYTHONHASHSEED
environment variable:

$PYTHONHASHSEED=42 python3.3
hashPython 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May  1 2012, 11:39:35)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> hash('a')
-5486454195961207828

$PYTHONHASHSEED=42 python3.3
Python 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May  1 2012, 11:39:35)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> hash('a')
-5486454195961207828

I'm still couldn't find how to actually get that seed, though.

Aaron Meurer

On Sat, May 26, 2012 at 10:58 PM, Aaron Meurer <asmeurer at gmail.com> wrote:
> Hi.
>
> First, I want to apologize if this is the wrong list for this
> question. If it is, kindly direct me to the correct one and I'll be on
> my way.
>
> I am attempting to "port" SymPy to Python 3.3. ?We already have full
> support for Python 3.2, but the new hash randomization in Python 3.3
> has brought up a bunch of new failures in our test suite. ?This is not
> a surprise, as a lot of operations in SymPy are dependent on hash
> values. ?But expecting the problem does not entirely help use to solve
> it.
>
> My question is this: is there a way to get the random seed used for
> hash randomization, and to print it out, and then to later input that
> seed to the interpreter? ?Without this ability, it becomes extremely
> difficult to reproduce test failures. ?The best bet is to restart the
> interpreter multiple times until it occurs. ?We already have
> experienced in SymPy several odd bugs that were caused by very rare
> seeds in the random module. ?This seed is easy to set, though, and we
> print it with each test run, so the handful of issues that have
> cropped up we've been able to fix. ?I'd like to have that same ability
> with hash randomization.
>
> I figured that this is a new feature, and people on this list are
> supposed to be experts on new Python features, so my hope is that
> either someone here will know the answer, or can point me in the right
> direction to one.
>
> If it is the case, as I fear/expect, that this is not possible, should
> I open an issue in the Python bug tracker? ?Should I raise the issue
> on python-dev? ?Has it been discussed before? ?I realize that there
> may be a security risk (imho a low one, though) in such a feature, so
> it might not be immediately accepted.
>
> Aaron Meurer

From martin at v.loewis.de  Sun May 27 08:22:27 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Sun, 27 May 2012 08:22:27 +0200
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
Message-ID: <20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>

> I'm still couldn't find how to actually get that seed, though.

In C, you can look at _Py_HashSecret. In Python, you need to write
an extension module, or use ctypes on the Python interpreter itself.

However, this is not the seed: when an RNG is used, there is no seed,
instead, the OS directly provides the hash secret. So your extension
would also have to support setting the secret, which is tricky because
the secret is already used by the time the extension gets loaded.

So you would have to change the interpreter to support such a feature.

If the crashes/test failures are frequent enough, I rather recommend
testing with PYTHONHASHSEED set to random integers.

Also: if a test fails due to hash randomization, it should normally
be possible to find the root cause by just reviewing the code (long
enough). It may not be possible to reproduce the failure, but it
should be obvious if a certain piece of code would fail under hash
randomization.

Regards,
Martin



From asmeurer at gmail.com  Sun May 27 10:07:55 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Sun, 27 May 2012 02:07:55 -0600
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
	<20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
Message-ID: <CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>

On Sun, May 27, 2012 at 12:22 AM,  <martin at v.loewis.de> wrote:
>> I'm still couldn't find how to actually get that seed, though.
>
>
> In C, you can look at _Py_HashSecret. In Python, you need to write
> an extension module, or use ctypes on the Python interpreter itself.
>
> However, this is not the seed: when an RNG is used, there is no seed,
> instead, the OS directly provides the hash secret. So your extension
> would also have to support setting the secret, which is tricky because
> the secret is already used by the time the extension gets loaded.
>
> So you would have to change the interpreter to support such a feature.
>
> If the crashes/test failures are frequent enough, I rather recommend
> testing with PYTHONHASHSEED set to random integers.

I see.  This would require spawning a new Python process, so it's not
ideal, but I guess it's the only solution.  I'll give ctypes a try
too.  I don't particularly feel like dabbling in C extensions just to
make our tests a little more helpful (maybe someone else more
courageous will give it a go).

>
> Also: if a test fails due to hash randomization, it should normally
> be possible to find the root cause by just reviewing the code (long
> enough). It may not be possible to reproduce the failure, but it
> should be obvious if a certain piece of code would fail under hash
> randomization.
>
> Regards,
> Martin

Ha!  Well, that's easy enough to say, but if all you have to work with
is an assertion that failed, and a very large code base, it might not
be so straight forward.  Furthermore, such situations are very often
not obvious (or else the author probably would not have written them
in the first place).

I do grant that this is possible in principle, but pragmatically
speaking, if it's possible to consistently reproduce a bug, it's 100
times easier to fix it.

It doesn't help that quite a few Python programmers don't understand
just what is and is not guaranteed by hash dependent objects.  For
example, I've seen this mistake made several times:

a = set(whatever) # or dict
b = list(a)
c = list(a)
assert b == c

The assertion does NOT have to hold, and I've seen situations where it doesn't.

That issue is pretty subtle.  The more common case is iterating
through a set or dict (or a tuple that was sorted by hash, which is
the most common case for SymPy), and there is some subtle fact about
the loop that makes the result differ depending on the result of
iteration.  Quite often, the result is still "correct" (in SymPy, this
generally means the answer is still mathematically correct), just not
the same as what the test expected.

Aaron Meurer

From martin at v.loewis.de  Sun May 27 10:24:19 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Sun, 27 May 2012 10:24:19 +0200
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
	<20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
	<CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
Message-ID: <20120527102419.Horde.Gu8uALuWis5PweSzHiUF8AA@webmail.df.eu>

> That issue is pretty subtle.  The more common case is iterating
> through a set or dict (or a tuple that was sorted by hash, which is
> the most common case for SymPy), and there is some subtle fact about
> the loop that makes the result differ depending on the result of
> iteration.  Quite often, the result is still "correct" (in SymPy, this
> generally means the answer is still mathematically correct), just not
> the same as what the test expected.

But there are standard procedures to deal with that very phenomenon:
use a proper equality function.

People have written tests for years that somehow relied on the order
of keys in a dictionary (an issue in particular for doctest). If you
find a failed assertion, and it involves an equality test, verify that
the comparison uses "normalized" representations of the value. If not,
add the normalization to all related test cases.

Regards,
Martin



From chris.jerdonek at gmail.com  Sun May 27 16:11:22 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sun, 27 May 2012 07:11:22 -0700
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
	<20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
	<CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
Message-ID: <CAOTb1wc59d0BrweB-_ZiBbFMc_0YgWPcBMtn=6mWr_uhaqiR5Q@mail.gmail.com>

On Sun, May 27, 2012 at 1:07 AM, Aaron Meurer <asmeurer at gmail.com> wrote:
> On Sun, May 27, 2012 at 12:22 AM, ?<martin at v.loewis.de> wrote:
>> Also: if a test fails due to hash randomization, it should normally
>> be possible to find the root cause by just reviewing the code (long
>> enough). It may not be possible to reproduce the failure, but it
>> should be obvious if a certain piece of code would fail under hash
>> randomization.
>>
> Ha! ?Well, that's easy enough to say, but if all you have to work with
> is an assertion that failed, and a very large code base, it might not
> be so straight forward. ?Furthermore, such situations are very often
> not obvious (or else the author probably would not have written them
> in the first place).

Sorry if this is obvious, but another suggestion is to include more
information in the assertion error message.  It's not clear from the
discussion so far if this is being done.  That way you can learn more
about the state of the code during the assertion failure.

I also spot checked a few spots in SymPy and see this being done frequently--

    def test_ratint():
        assert ratint(S(0), x) == 0

Options include using the extended form of the assert statement, or
the msg parameter of unittest's various assertion methods.

--Chris

From solipsis at pitrou.net  Sun May 27 23:03:07 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 27 May 2012 21:03:07 +0000 (UTC)
Subject: [Python-porting] Control of hash randomization
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
	<20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
	<CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
Message-ID: <loom.20120527T230033-620@post.gmane.org>

Aaron Meurer <asmeurer at ...> writes:
> 
> It doesn't help that quite a few Python programmers don't understand
> just what is and is not guaranteed by hash dependent objects.  For
> example, I've seen this mistake made several times:
> 
> a = set(whatever) # or dict
> b = list(a)
> c = list(a)
> assert b == c
> 
> The assertion does NOT have to hold, and I've seen situations where it
> doesn't.

Actually, it should hold, since the iteration order of a given set or dict 
doesn't change (similarly, dict.keys(), dict.values() and dict.items() should 
all iterate in the same order). However, constructing the set in different ways 
may lead to different iteration orders.

Regards

Antoine.



From asmeurer at gmail.com  Sun May 27 23:40:27 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Sun, 27 May 2012 15:40:27 -0600
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <loom.20120527T230033-620@post.gmane.org>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
	<20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
	<CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
	<loom.20120527T230033-620@post.gmane.org>
Message-ID: <CAKgW=6K6RNhcPw3AU53PhxkwHgC3rBsBF6mJwmxjHeJ_Jk0pTQ@mail.gmail.com>

On Sun, May 27, 2012 at 3:03 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Aaron Meurer <asmeurer at ...> writes:
>>
>> It doesn't help that quite a few Python programmers don't understand
>> just what is and is not guaranteed by hash dependent objects. ?For
>> example, I've seen this mistake made several times:
>>
>> a = set(whatever) # or dict
>> b = list(a)
>> c = list(a)
>> assert b == c
>>
>> The assertion does NOT have to hold, and I've seen situations where it
>> doesn't.
>
> Actually, it should hold, since the iteration order of a given set or dict
> doesn't change (similarly, dict.keys(), dict.values() and dict.items() should
> all iterate in the same order). However, constructing the set in different ways
> may lead to different iteration orders.
>
> Regards
>
> Antoine.

Yes, that was a simplification of the problem.  I think when I saw it
some very subtle thing was happening to the set.  I can look it up if
anyone's interested.

Aaron Meurer

From martin at v.loewis.de  Mon May 28 00:19:54 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Mon, 28 May 2012 00:19:54 +0200
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <CAKgW=6K6RNhcPw3AU53PhxkwHgC3rBsBF6mJwmxjHeJ_Jk0pTQ@mail.gmail.com>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
	<20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
	<CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
	<loom.20120527T230033-620@post.gmane.org>
	<CAKgW=6K6RNhcPw3AU53PhxkwHgC3rBsBF6mJwmxjHeJ_Jk0pTQ@mail.gmail.com>
Message-ID: <20120528001954.Horde.dbNCJdjz9kRPwqiKYL4EShA@webmail.df.eu>


Zitat von Aaron Meurer <asmeurer at gmail.com>:

> On Sun, May 27, 2012 at 3:03 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> Aaron Meurer <asmeurer at ...> writes:
>>>
>>> It doesn't help that quite a few Python programmers don't understand
>>> just what is and is not guaranteed by hash dependent objects. ?For
>>> example, I've seen this mistake made several times:
>>>
>>> a = set(whatever) # or dict
>>> b = list(a)
>>> c = list(a)
>>> assert b == c
>>>
>>> The assertion does NOT have to hold, and I've seen situations where it
>>> doesn't.
>>
>> Actually, it should hold, since the iteration order of a given set or dict
>> doesn't change (similarly, dict.keys(), dict.values() and  
>> dict.items() should
>> all iterate in the same order). However, constructing the set in  
>> different ways
>> may lead to different iteration orders.
>>
>> Regards
>>
>> Antoine.
>
> Yes, that was a simplification of the problem.  I think when I saw it
> some very subtle thing was happening to the set.  I can look it up if
> anyone's interested.

I was puzzled by your original claim as well. If you "admit" that the
set might have been touched somehow, I can readily believe that it gives
elements in a different order.

Regards,
Martin




From asmeurer at gmail.com  Mon May 28 00:49:31 2012
From: asmeurer at gmail.com (Aaron Meurer)
Date: Sun, 27 May 2012 16:49:31 -0600
Subject: [Python-porting] Control of hash randomization
In-Reply-To: <CAOTb1wc59d0BrweB-_ZiBbFMc_0YgWPcBMtn=6mWr_uhaqiR5Q@mail.gmail.com>
References: <CAKgW=6Lpn-UGHUEh1r0Wx1_BRqtbx5EmGVsgf6fG7ft-7PpbvA@mail.gmail.com>
	<CAKgW=6LZj6Qz67LhKa-1QkX56naOob2yeUGaRt61ZP3zoCdNAA@mail.gmail.com>
	<20120527082227.Horde.zg_8Y1NNcXdPwcgjvV0QXMA@webmail.df.eu>
	<CAKgW=6+Cbms+FeG1p-WNgPSPwBLZf_aLj_gMnf4-9hWmcC97tQ@mail.gmail.com>
	<CAOTb1wc59d0BrweB-_ZiBbFMc_0YgWPcBMtn=6mWr_uhaqiR5Q@mail.gmail.com>
Message-ID: <CAKgW=6K9CyzJdJ_Oq=VEM79nRON4rnRq0OGpV=t_AeGf3C4mKQ@mail.gmail.com>

On Sun, May 27, 2012 at 2:24 AM,  <martin at v.loewis.de> wrote:
> But there are standard procedures to deal with that very phenomenon:
> use a proper equality function.
>
> People have written tests for years that somehow relied on the order
> of keys in a dictionary (an issue in particular for doctest). If you
> find a failed assertion, and it involves an equality test, verify that
> the comparison uses "normalized" representations of the value. If not,
> add the normalization to all related test cases.
>
> Regards,
> Martin

I appreciate what you're trying to say here, but you need to
understand that in SymPy, the result of a test is a mathematical
expression (in the most literal sense).  A "normalization" function in
this case would be a simplification function.  Expression
simplification is very difficult.  Expression normalization is even
more difficult.  Technically speaking, for a large enough class of
functions, it's impossible.  I can point you to papers that
demonstrate why expression simplification/normalization is so
difficult if you are interested.

In SymPy, we have many tests like

    assert solve(3*x+5+2**(-5*x+3), x) in [
        [-((25*log(2) - 3*LambertW(-10240*2**(Rational(1,
3))*log(2)/3))/(15*log(2)))],
        [Rational(-5, 3) + LambertW(log(2**(-10240*2**(Rational(1,
3))/3)))/(5*log(2))],
        [-Rational(5,3) +
LambertW(-10240*2**Rational(1,3)*log(2)/3)/(5*log(2))],
        [(-25*log(2) + 3*LambertW(-10240*2**(Rational(1,
3))*log(2)/3))/(15*log(2))],
        [-((25*log(2) - 3*LambertW(-10240*2**(Rational(1,
3))*log(2)/3)))/(15*log(2))],
        [-(25*log(2) - 3*LambertW(log(2**(-10240*2**Rational(1,
3)/3))))/(15*log(2))],
        [(25*log(2) - 3*LambertW(log(2**(-10240*2**Rational(1,
3)/3))))/(-15*log(2))]
        ]

This was note the test writer trying to come up with every possible
form of the expression.  At some point in time, solve() returned each
one of those answers, but some subtle change in the code somewhere
resulted in it returning a different one.  Actually, in this
particular case, we probably could normalize by expanding, but that's
a simple case.  Here's a more advanced one (heurisch performs symbolic
integration):

assert heurisch(sin(x)*cos(x), x) in [sin(x)**2 / 2, -cos(x)**2 / 2]

This is a classic example from calculus of a function the produces a
different result depending on how you integrate it.  These two
functions are not mathematically equal.  That's because heurisch() is
only guaranteed to produce an antiderivative, which can differ by a
constant.

Even if simplify() were reliable, writing "assert simplify(a - b) ==
0" instead of "assert a == b" would make our tests much slower, and
would add the possibility that any test failure anywhere is the result
of a bug in simplify().

What we really need to do is change our internal ordering to not use
hashes, and to avoid iterating through a dictionary when the result
could be different (but still mathematically the same).  At least hash
randomization will point us to the places where this is happening.

On Sun, May 27, 2012 at 8:11 AM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> On Sun, May 27, 2012 at 1:07 AM, Aaron Meurer <asmeurer at gmail.com> wrote:
>> On Sun, May 27, 2012 at 12:22 AM, ?<martin at v.loewis.de> wrote:
>>> Also: if a test fails due to hash randomization, it should normally
>>> be possible to find the root cause by just reviewing the code (long
>>> enough). It may not be possible to reproduce the failure, but it
>>> should be obvious if a certain piece of code would fail under hash
>>> randomization.
>>>
>> Ha! ?Well, that's easy enough to say, but if all you have to work with
>> is an assertion that failed, and a very large code base, it might not
>> be so straight forward. ?Furthermore, such situations are very often
>> not obvious (or else the author probably would not have written them
>> in the first place).
>
> Sorry if this is obvious, but another suggestion is to include more
> information in the assertion error message. ?It's not clear from the
> discussion so far if this is being done. ?That way you can learn more
> about the state of the code during the assertion failure.
>
> I also spot checked a few spots in SymPy and see this being done frequently--
>
> ? ?def test_ratint():
> ? ? ? ?assert ratint(S(0), x) == 0
>
> Options include using the extended form of the assert statement, or
> the msg parameter of unittest's various assertion methods.
>
> --Chris

Yes, this is basically how we do our tests, "assert
result_of_some_computation == the_correct_answer".  The solution in
this case is for us to be using py.test instead of our home-grown test
runner, which provides this output for us.

Aaron Meurer