From barry at barrys-emacs.org  Fri Dec  5 21:15:45 2008
From: barry at barrys-emacs.org (Barry Scott)
Date: Fri, 5 Dec 2008 20:15:45 +0000
Subject: [Python-porting] PyCXX has been ported to Python 3
Message-ID: <37EDA024-C42C-4A01-BD05-3D1868C557B9@barrys-emacs.org>

I have been working on supporting Python 2 and Python 3 for PyCXX.

Python 2 will be supported from

	svn: https://cxx.svn.sourceforge.net/svnroot/cxx/branches/pycxx-5-maint/CXX

and will be release as V5.6

Python 3 will be supported from

	svn: https://cxx.svn.sourceforge.net/svnroot/cxx/trunk/CXX

and will be release as V6.0

It is possible to compile wth -DPYCXX_2TO3 defined to enable one set of
source to be compiled against Python 2 with V5.6 or Python 3 with V6.0.

As a proof of concept pysvn on trunk can be built against Python 2 or  
Python 3.

Barry

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20081205/f09d2902/attachment.htm>

From brett at python.org  Fri Dec  5 22:58:28 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Dec 2008 13:58:28 -0800
Subject: [Python-porting] PyCXX has been ported to Python 3
In-Reply-To: <37EDA024-C42C-4A01-BD05-3D1868C557B9@barrys-emacs.org>
References: <37EDA024-C42C-4A01-BD05-3D1868C557B9@barrys-emacs.org>
Message-ID: <bbaeab100812051358v1f41a33ajcec322a09fe8195e@mail.gmail.com>

On Fri, Dec 5, 2008 at 12:15, Barry Scott <barry at barrys-emacs.org> wrote:
> I have been working on supporting Python 2 and Python 3 for PyCXX.
> Python 2 will be supported from
> svn: https://cxx.svn.sourceforge.net/svnroot/cxx/branches/pycxx-5-maint/CXX
> and will be release as V5.6
> Python 3 will be supported from
> svn: https://cxx.svn.sourceforge.net/svnroot/cxx/trunk/CXX
> and will be release as V6.0
> It is possible to compile wth -DPYCXX_2TO3 defined to enable one set of
> source to be compiled against Python 2 with V5.6 or Python 3 with V6.0.
> As a proof of concept pysvn on trunk can be built against Python 2 or Python
> 3.

Very cool!

-Brett

From theaney at gmail.com  Sat Dec  6 03:35:08 2008
From: theaney at gmail.com (Tim Heaney)
Date: Fri, 5 Dec 2008 21:35:08 -0500
Subject: [Python-porting] News interface?
Message-ID: <d78741030812051835n45270bfen1114c5cfa5da4439@mail.gmail.com>

Is this list going to be available via Gmane?

  http://gmane.org/subscribe.php

Thanks!

Tim

From martin at v.loewis.de  Sat Dec  6 11:31:52 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 06 Dec 2008 11:31:52 +0100
Subject: [Python-porting] Port of psycopg2
Message-ID: <493A5498.2070602@v.loewis.de>

I created to run psycopg2 on Python 3.0, at

http://www.dcl.hpi.uni-potsdam.de/home/loewis/psycopg_3k.diff

I also submitted the patch to Frederico di Gregorio, who will
integrate it when he finds time.

The patch passes nearly all tests (the failures seem unrelated,
since the unmodified code also fails in nearly the same way).

Regards,
Martin

From georg at python.org  Sat Dec  6 13:05:07 2008
From: georg at python.org (Georg Brandl)
Date: Sat, 06 Dec 2008 13:05:07 +0100
Subject: [Python-porting] News interface?
In-Reply-To: <d78741030812051835n45270bfen1114c5cfa5da4439@mail.gmail.com>
References: <d78741030812051835n45270bfen1114c5cfa5da4439@mail.gmail.com>
Message-ID: <493A6A73.6030602@python.org>

Tim Heaney schrieb:
> Is this list going to be available via Gmane?
> 
>   http://gmane.org/subscribe.php

Yes, I'll handle adding the subscription to Gmane.

Georg

From paul_hildebrandt at yahoo.com  Sat Dec  6 21:23:11 2008
From: paul_hildebrandt at yahoo.com (Paul Hildebrandt)
Date: Sat, 6 Dec 2008 12:23:11 -0800 (PST)
Subject: [Python-porting] Porting Tools
Message-ID: <121095.92675.qm@web111507.mail.gq1.yahoo.com>

The first step in porting to 3.0 is to come up with a comprehensive set of tests.  

The Pythoscope (http://pythoscope.org/) project mission statement is:
To create an easily customizable and extensible open source tool that will automatically, or semi-automatically, generate unit tests for legacysystems written in Python.

It's a hard problem and it's not perfect by any means but it's coming along.   It may help you out.  

Paul



      

From brett at python.org  Sat Dec  6 23:12:59 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 6 Dec 2008 14:12:59 -0800
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <493A5498.2070602@v.loewis.de>
References: <493A5498.2070602@v.loewis.de>
Message-ID: <bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>

On Sat, Dec 6, 2008 at 02:31, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> I created to run psycopg2 on Python 3.0, at
>
> http://www.dcl.hpi.uni-potsdam.de/home/loewis/psycopg_3k.diff
>
> I also submitted the patch to Frederico di Gregorio, who will
> integrate it when he finds time.
>
> The patch passes nearly all tests (the failures seem unrelated,
> since the unmodified code also fails in nearly the same way).
>

Martin, the porting machine!

I know this keeps coming up, but I have not heard of a porting doc
being written yet. Is there one somewhere like the wiki?

-Brett

From georg at python.org  Sat Dec  6 23:22:18 2008
From: georg at python.org (Georg Brandl)
Date: Sat, 06 Dec 2008 23:22:18 +0100
Subject: [Python-porting] Number of Python 3 packages
Message-ID: <493AFB1A.40609@python.org>

Hi,

this page lists all PyPI packages with the "Python 3" trove classifier:

http://pypi.python.org/pypi?:action=browse&c=533&show=all

I wrote a quick script that displays the number of packages versus time,
and puts the graph at

http://dev.pocoo.org/~gbrandl/py3pkgs.png

(It's not very interesting at the moment...)

Georg

From georg at python.org  Sat Dec  6 23:34:31 2008
From: georg at python.org (Georg Brandl)
Date: Sat, 06 Dec 2008 23:34:31 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
References: <493A5498.2070602@v.loewis.de>
	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
Message-ID: <493AFDF7.90006@python.org>

Brett Cannon schrieb:
> On Sat, Dec 6, 2008 at 02:31, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> I created to run psycopg2 on Python 3.0, at
>>
>> http://www.dcl.hpi.uni-potsdam.de/home/loewis/psycopg_3k.diff
>>
>> I also submitted the patch to Frederico di Gregorio, who will
>> integrate it when he finds time.
>>
>> The patch passes nearly all tests (the failures seem unrelated,
>> since the unmodified code also fails in nearly the same way).
>>
> 
> Martin, the porting machine!
> 
> I know this keeps coming up, but I have not heard of a porting doc
> being written yet. Is there one somewhere like the wiki?

I searched for "Porting", and found only Martin's PortingDjango page.

We should start drafting a porting document, which I also would like
to include in the official docs as soon as it's in a good shape, in
the Wiki.

Any suggestions for the general lay-out of such a document?

Georg


From martin at v.loewis.de  Sun Dec  7 00:44:07 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 07 Dec 2008 00:44:07 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
References: <493A5498.2070602@v.loewis.de>
	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
Message-ID: <493B0E47.4030105@v.loewis.de>

> Martin, the porting machine!

:-)

> I know this keeps coming up, but I have not heard of a porting doc
> being written yet. Is there one somewhere like the wiki?

I know this wasn't directed at me - I'm not very good at writing
new docs (although I can add to existing docs well).

I vaguely recall somebody reporting collecting and writing documentation
for porting extension modules, but I forgot the details. In any case,
here is what I found in psycopg:

- bytes vs. strings 1. Design decisions needs to be taken what exactly
  must be represented as bytes, and what as strings. In many cases, that
  was easy for psycopg, except for the question how SQL queries are
  represented. It appears clear that they are plain text, however, the
  Postgres API requires them to be transmitted in the connection
  encoding. I still decided to represent them internally in Unicode,
  but converting them to the connection encoding as early as possible
  probably would have worked as well.

- bytes vs. strings 2. To keep the implementation portable across 2.x
  and 3.x, I added a number of macros. Most useful was Text_FromUTF8,
  particularly when applied to the (many) string literals. It is
  defined as PyString_FromString for 2.x, and PyUnicode_FromString for
  3.x.

- RO is gone, you need to use READONLY (which also works in 2.x).

- PyVarObject_HEAD_INIT needs to be used for types. I define it for
  2.x if it isn't already defined.

- the buffer object is gone; I use memoryview in 3.x.

- various tests where in the code of the form
  if version_major ==  and version_minor > 4 (say, or 5)
  This will break for 3.x; you have to write
  if (version_major == 2 and version_minor > 4) or version_major > 2

- module initialization is different. I moved the majority of the
  code into a static function, which then gets conditionally called from
  either the 2.x or 3.x init routine.

- Python code 1: I used the 2to3 support in distutils

- Python code 2: setup.py needs to run in both versions. I had to
  replace popen2 with subprocess if available. Also, map() now
  returns an iterator, which I explicitly convert into list, and
  so on.

- Python code 3: the test suite doesn't get installed, and hence
  not auto-converted with 2to3 support. I explicitly added a 2to3
  conversion into the test runner, which copies the py3 version of
  the test into a separate directory.

Regards,
Martin

From brett at python.org  Sun Dec  7 01:09:18 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 6 Dec 2008 16:09:18 -0800
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <493AFDF7.90006@python.org>
References: <493A5498.2070602@v.loewis.de>
	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
	<493AFDF7.90006@python.org>
Message-ID: <bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>

On Sat, Dec 6, 2008 at 14:34, Georg Brandl <georg at python.org> wrote:
> Brett Cannon schrieb:
>> On Sat, Dec 6, 2008 at 02:31, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> I created to run psycopg2 on Python 3.0, at
>>>
>>> http://www.dcl.hpi.uni-potsdam.de/home/loewis/psycopg_3k.diff
>>>
>>> I also submitted the patch to Frederico di Gregorio, who will
>>> integrate it when he finds time.
>>>
>>> The patch passes nearly all tests (the failures seem unrelated,
>>> since the unmodified code also fails in nearly the same way).
>>>
>>
>> Martin, the porting machine!
>>
>> I know this keeps coming up, but I have not heard of a porting doc
>> being written yet. Is there one somewhere like the wiki?
>
> I searched for "Porting", and found only Martin's PortingDjango page.
>

Thought so.

> We should start drafting a porting document,

Definitely.

> which I also would like
> to include in the official docs

Yep.

> as soon as it's in a good shape, in
> the Wiki.
>
> Any suggestions for the general lay-out of such a document?

Well, there are two things that require porting: Python and extension
modules. So that is a high-level point of separation. From there we
can list the various steps that are needed (i.e. port to 2.6 with no
warnings, run with -3, add any __future__ statements you want, run
2to3, test). The trick is where to put little tips like making sure
you clearly separate your string code from your bytes code.

-Brett

From georg at python.org  Sun Dec  7 13:34:20 2008
From: georg at python.org (Georg Brandl)
Date: Sun, 07 Dec 2008 13:34:20 +0100
Subject: [Python-porting] Porting guides in the Wiki
Message-ID: <493BC2CC.8030404@python.org>

Hi,

I've created two porting guide skeletons in the Wiki. The portal page

  http://wiki.python.org/moin/PortingToPy3k

links to one page for Python code, one page for C code.  The C code one
is currently a copy of the doc Benjamin wrote (which I completely forgot
about).  It can serve as a good starting point.

For the Python code one, I've just started a list of topics and pasted
Martin's comments from psycopg2 porting.

Perhaps we should also collect blog posts from module authors porting
their modules?

Some notes for editors:

* Please keep the formatting as reST, so that the finished docs can easily
  be put into the official docs when ready.

* Use Sphinx roles if you know them; this will make proper marking up
  faster.  To make the stock docutils not complain about Sphinx roles
  in the text, put a role directive at the top of the page for each
  Sphinx role you introduce.  For example, for the :func: role, add

  .. role:: func(literal)

* Otherwise, please don't feel restricted by anything.  Feel free to add
  just notes, comments, links or whatever -- it is a wiki after all.

cheers,
Georg

From musiccomposition at gmail.com  Sun Dec  7 16:15:22 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sun, 7 Dec 2008 09:15:22 -0600
Subject: [Python-porting] Porting guides in the Wiki
In-Reply-To: <493BC2CC.8030404@python.org>
References: <493BC2CC.8030404@python.org>
Message-ID: <1afaf6160812070715nbb26ab0yfb267de639e05643@mail.gmail.com>

On Sun, Dec 7, 2008 at 6:34 AM, Georg Brandl <georg at python.org> wrote:
> Hi,
>
> I've created two porting guide skeletons in the Wiki. The portal page

Excellent! Maybe we should all write blog posts advertising them?

>
>  http://wiki.python.org/moin/PortingToPy3k
>
> links to one page for Python code, one page for C code.  The C code one
> is currently a copy of the doc Benjamin wrote (which I completely forgot
> about).  It can serve as a good starting point.

It's definitely only a starting point. At the moment, it consists of
some crummy examples with minimal explanations.

>
> For the Python code one, I've just started a list of topics and pasted
> Martin's comments from psycopg2 porting.

With regards to the 2to3 portion of Python porting, is it sufficient
to link to the current 2to3 docs? Or should those actually be
integrated into the porting document?

>
> Perhaps we should also collect blog posts from module authors porting
> their modules?

It would be nice to have more real world examples like Martin's.




-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From skippy.hammond at gmail.com  Sun Dec  7 22:45:54 2008
From: skippy.hammond at gmail.com (Mark Hammond)
Date: Mon, 08 Dec 2008 08:45:54 +1100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>
	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>
Message-ID: <493C4412.9050004@gmail.com>

Brett Cannon wrote:
> Well, there are two things that require porting: Python and extension
> modules. So that is a high-level point of separation. From there we
> can list the various steps that are needed (i.e. port to 2.6 with no
> warnings, run with -3, add any __future__ statements you want, run
> 2to3, test). The trick is where to put little tips like making sure
> you clearly separate your string code from your bytes code.

Although the official line is to port to 2.6 first, it will be 
interesting to know how many people who are hanging out here care about 
earlier versions too?  ie, while many *application* authors can easily 
choose to target a specific version, *library* authors may not be so lucky.

Is there anyone else here for whom the step of "port to python 2.6" is a 
deal-breaker?

Mark


From musiccomposition at gmail.com  Sun Dec  7 23:53:41 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sun, 7 Dec 2008 16:53:41 -0600
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <493C4412.9050004@gmail.com>
References: <493A5498.2070602@v.loewis.de>
	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
	<493AFDF7.90006@python.org>
	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>
	<493C4412.9050004@gmail.com>
Message-ID: <1afaf6160812071453r18bb2561u483b436a672bb9db@mail.gmail.com>

On Sun, Dec 7, 2008 at 3:45 PM, Mark Hammond <skippy.hammond at gmail.com> wrote:
> Brett Cannon wrote:
>>
>> Well, there are two things that require porting: Python and extension
>> modules. So that is a high-level point of separation. From there we
>> can list the various steps that are needed (i.e. port to 2.6 with no
>> warnings, run with -3, add any __future__ statements you want, run
>> 2to3, test). The trick is where to put little tips like making sure
>> you clearly separate your string code from your bytes code.
>
> Although the official line is to port to 2.6 first, it will be interesting
> to know how many people who are hanging out here care about earlier versions
> too?  ie, while many *application* authors can easily choose to target a
> specific version, *library* authors may not be so lucky.
>
> Is there anyone else here for whom the step of "port to python 2.6" is a
> deal-breaker?

I'm not myself dealing with this, but Armin Ronacher and I were
chatting about this on IRC. He says that for Jinja it is impossible to
maintain 2.x and 3.x in the same code base while keeping compatibility
with 2.4. The result of this conversation is a 2to3 feature request:
Add pragmas to 2to3 that would hint at the correct translation.



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From python at mikewatkins.ca  Mon Dec  8 01:25:13 2008
From: python at mikewatkins.ca (Michael Watkins)
Date: Sun, 7 Dec 2008 16:25:13 -0800 (PST)
Subject: [Python-porting] A Python 2 *and* 3 success story
Message-ID: <4665.24.84.177.146.1228695913.squirrel@webmail.solutionroute.ca>

David Binger of the MEMS Nanotechnology Exchange (MNX) writes about their
experience migrating their public and private code base such that it can
run on either Python >= 2.4 or 3.x deployment targets:

    http://mail.mems-exchange.org/durusmail/qp/441/

I'm a user of the packages, not a developer, and have been using QP and
Durus since they first were born, and Quixote before that. I was happy to
see that the "compatibility shims" do not feel too ugly nor do they seem
to overly intrude on the code. There doesn't seem to be any downside for
*this* particular set of packages to support Python >= 2.4 (which includes
3.x) out of the same code base.

You find navigate down through the packages and end up at a "browse"
interface:

    http://www.mems-exchange.org/software/

I helped do testing of QP and related packages and through that experience
noted that of the various changes, probably the metaclass syntax change
had the potential for the ugliest 2-and-3 style workaround but in the end
there was a slightly more graceful solution staring me in the face that
works on both major Python versions:

    http://mikewatkins.ca/2008/11/29/python-2-and-3-metaclasses/





From mal at egenix.com  Mon Dec  8 10:46:57 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 08 Dec 2008 10:46:57 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
References: <493A5498.2070602@v.loewis.de>
	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
Message-ID: <493CED11.1090702@egenix.com>

On 2008-12-06 23:12, Brett Cannon wrote:
> On Sat, Dec 6, 2008 at 02:31, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> I created to run psycopg2 on Python 3.0, at
>>
>> http://www.dcl.hpi.uni-potsdam.de/home/loewis/psycopg_3k.diff
>>
>> I also submitted the patch to Frederico di Gregorio, who will
>> integrate it when he finds time.
>>
>> The patch passes nearly all tests (the failures seem unrelated,
>> since the unmodified code also fails in nearly the same way).
>>
> 
> Martin, the porting machine!
> 
> I know this keeps coming up, but I have not heard of a porting doc
> being written yet. Is there one somewhere like the wiki?

FWIW: I posted a list of notes from our wiki some days ago:

http://mail.python.org/pipermail/python-porting/2008-November/000000.html

Good to see some action on this list :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 08 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2008-12-02: Released mxODBC.Connect 1.0.0      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Mon Dec  8 10:52:26 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 08 Dec 2008 10:52:26 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <493CED11.1090702@egenix.com>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
	<493CED11.1090702@egenix.com>
Message-ID: <493CEE5A.4020800@egenix.com>

On 2008-12-08 10:46, M.-A. Lemburg wrote:
> On 2008-12-06 23:12, Brett Cannon wrote:
>> On Sat, Dec 6, 2008 at 02:31, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> I created to run psycopg2 on Python 3.0, at
>>>
>>> http://www.dcl.hpi.uni-potsdam.de/home/loewis/psycopg_3k.diff
>>>
>>> I also submitted the patch to Frederico di Gregorio, who will
>>> integrate it when he finds time.
>>>
>>> The patch passes nearly all tests (the failures seem unrelated,
>>> since the unmodified code also fails in nearly the same way).
>>>
>> Martin, the porting machine!
>>
>> I know this keeps coming up, but I have not heard of a porting doc
>> being written yet. Is there one somewhere like the wiki?
> 
> FWIW: I posted a list of notes from our wiki some days ago:
> 
> http://mail.python.org/pipermail/python-porting/2008-November/000000.html
> 
> Good to see some action on this list :-)

We've since added these additional notes:

"""
== Notes for Python modules and package ==

 * Standard library was reorganized.

 See [http://www.python.org/dev/peps/pep-3108/ PEP 3108] for all the details.

 It's probably best to let 2to3 do the job of renaming imports.

 A couple of modules were also removed, e.g. the rfc822, popen2, urllib,
htmllib, UserList, UserDict, UserString modules. Their functionality is usually
available in one of the other more recent modules (e.g. the email package) or
via type subclassing.

 Others were removed without replacements, e.g. imputil, the string functions
wrapper (stringold), sgmllib.

 * new module is gone and so is types.ClassType

 To create new-style classes, you can use type() instead, e.g.
 {{{
myclass = type("MyClass", (object,), classdict)
}}}

 The above also works in Python 2.3 and onwards, so new code should start to be
written this way.

== Notes for C extensions ==

 * PyString_*() API have been renamed to PyBytes_*() and some were removed

 Check whether PyBytes_*() APIs and objects are really appropriate for the task
and switch to PyUnicode_*() APIs as necessary.

 * PyExc_StandardError no longer exists

 It should be replaced with PyExc_Exception.

 See [http://python.org/dev/peps/pep-0352/ PEP 352].

 * PyInt_*() APIs no longer exist.

 Python 3 uses the Python long implementation to store and handle integers.

 Use PyLong_*() APIs instead.

 PyInt_Check() uses should be either removed or changed to PyLong_Check().

 * Extension module initialization has changed completely in Python 3.

 See the
[http://docs.python.org/howto/cporting.html#module-initialization-and-state
porting howto] for an overview.

 Unfortunately, the Python 3 C API docs don't appear to have been updated to
this new method.

 * Type object initialization has changed.

 See [http://www.python.org/dev/peps/pep-3123/ PEP 3123] for details.

 You now have to write {{{
static PyTypeObject MyType = {
        PyVarObject_HEAD_INIT(NULL, 0)
        "MyType",
        ...
}
}}}
 instead of the version commonly found in Python 2 extensions: {{{
static PyTypeObject MyType = {
        PyObject_HEAD_INIT(&PyType_Type)
        0,                                      /* ob_size */
        "MyType",
        ...
}
}}}

 * The type slot interface has changed significantly.

 PyNumberMethods have changed due to removal of the division, oct, hex and
coercion slots.

 PySequenceMethods have changed, but maintained binary compatibility by
replacing the removed slice functions with dummy pointers.

 PyBufferProcs is a completely new design.

 A lot type flags were removed.

Questions:

 * The buffer object is gone in Python 3 and was replaced with a memoryview object.

 The memoryview object is just a wrapper around the low-level Py_buffer C struct
that represents buffers in Python 3.

 Strange enough, it is not clear how this can be used to efficiently access the
memory that the Py_buffer is maintaining. There are no macros defined for this
and the documentation points straight to the C struct members.

 [http://docs.python.org/dev/3.0/c-api/buffer.html Buffer documentation].

 The [http://www.python.org/dev/peps/pep-3118/ PEP 3118] describes the new API
in more detail and also gives a few examples.

 Overall, the whole buffer protocol design seems to have went overboard. We may
need to code our own little helper ? la PyObject_AsReadBuffer() in Python 2.

"""

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 08 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2008-12-02: Released mxODBC.Connect 1.0.0      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Mon Dec  8 13:46:39 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 08 Dec 2008 13:46:39 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <493C4412.9050004@gmail.com>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>
	<493C4412.9050004@gmail.com>
Message-ID: <493D172F.6030304@egenix.com>

On 2008-12-07 22:45, Mark Hammond wrote:
> Brett Cannon wrote:
>> Well, there are two things that require porting: Python and extension
>> modules. So that is a high-level point of separation. From there we
>> can list the various steps that are needed (i.e. port to 2.6 with no
>> warnings, run with -3, add any __future__ statements you want, run
>> 2to3, test). The trick is where to put little tips like making sure
>> you clearly separate your string code from your bytes code.
> 
> Although the official line is to port to 2.6 first, it will be
> interesting to know how many people who are hanging out here care about
> earlier versions too?  ie, while many *application* authors can easily
> choose to target a specific version, *library* authors may not be so lucky.
> 
> Is there anyone else here for whom the step of "port to python 2.6" is a
> deal-breaker?

I'm not sure where the problems would lie in porting to Python 2.6
first.

The latest big porting effort was moving from Python 2.4 to 2.5
due to the Py_ssize_t changes, but those only affected C extensions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 08 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2008-12-02: Released mxODBC.Connect 1.0.0      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mhammond at skippinet.com.au  Mon Dec  8 14:46:43 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Tue, 9 Dec 2008 00:46:43 +1100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <493D172F.6030304@egenix.com>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>
	<493C4412.9050004@gmail.com> <493D172F.6030304@egenix.com>
Message-ID: <079701c9593b$6c7b6b20$45724160$@com.au>

> > Is there anyone else here for whom the step of "port to python 2.6"
> is a deal-breaker?
> 
> I'm not sure where the problems would lie in porting to Python 2.6
> first.

It's more about moving to Python 2.6 syntax *only*.  eg, using 'bytes' or
memory views etc as porting aids isn't that useful when targetting earlier
versions.  Further, I meant "deal breaker" in a figurative sense - ie, not a
real blocker, just an extra complication than what the "general" porting
guide might bother considering - but by implication, something that might be
worth addressing *somewhere*; eg, I'm finding '.encode('ascii')' a
reasonable porting aid for bytes objects in many 'demo' byte literal cases,
such as appending \0 chars, etc.

'Cheers'.encode('ascii'), ly,

Mark


From brett at python.org  Mon Dec  8 19:58:28 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 8 Dec 2008 10:58:28 -0800
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <079701c9593b$6c7b6b20$45724160$@com.au>
References: <493A5498.2070602@v.loewis.de>
	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
	<493AFDF7.90006@python.org>
	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>
	<493C4412.9050004@gmail.com> <493D172F.6030304@egenix.com>
	<079701c9593b$6c7b6b20$45724160$@com.au>
Message-ID: <bbaeab100812081058g3bc62e9cw54cfd694e6bfd335@mail.gmail.com>

On Mon, Dec 8, 2008 at 05:46, Mark Hammond <mhammond at skippinet.com.au> wrote:
>> > Is there anyone else here for whom the step of "port to python 2.6"
>> is a deal-breaker?
>>
>> I'm not sure where the problems would lie in porting to Python 2.6
>> first.
>
> It's more about moving to Python 2.6 syntax *only*.  eg, using 'bytes' or
> memory views etc as porting aids isn't that useful when targetting earlier
> versions.  Further, I meant "deal breaker" in a figurative sense - ie, not a
> real blocker, just an extra complication than what the "general" porting
> guide might bother considering - but by implication, something that might be
> worth addressing *somewhere*; eg, I'm finding '.encode('ascii')' a
> reasonable porting aid for bytes objects in many 'demo' byte literal cases,
> such as appending \0 chars, etc.

Should we consider recommending that people who are willing to put in
the time to switch to unicode for strings and str for bytes if they
need to support < 2.6? Obviously a bit of work but it will make the
distinction clear and allow support for earlier Python versions.

-Brett

From theller at ctypes.org  Mon Dec  8 21:26:34 2008
From: theller at ctypes.org (Thomas Heller)
Date: Mon, 08 Dec 2008 21:26:34 +0100
Subject: [Python-porting] Some questions about build_py_2to3
Message-ID: <493D82FA.1080709@ctypes.org>

I have some beginners questions about distutils build_py_2to3 command:

- Can it convert doctests?  How?

- Shouldn't build_py_2to3 use a differnt build directory than the normal
build_py command?  Perhaps build/lib-3 or something like that?

- Should programming idioms that it fails to convert be reported as bugs?
Here is one example that isn't converted at all:

  class CoClass(COMObject):
      from comtypes._meta import _coclass_meta as __metaclass__


It is not obvious immeditately how the converted code should look like,
but at least the converter could issue a warning.

-- 
Thanks,
Thomas

From regebro at gmail.com  Mon Dec  8 21:48:59 2008
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 8 Dec 2008 21:48:59 +0100
Subject: [Python-porting] Some questions about build_py_2to3
In-Reply-To: <493D82FA.1080709@ctypes.org>
References: <493D82FA.1080709@ctypes.org>
Message-ID: <319e029f0812081248i439b434dpa6c718a7d8b22449@mail.gmail.com>

On Mon, Dec 8, 2008 at 21:26, Thomas Heller <theller at ctypes.org> wrote:
> Here is one example that isn't converted at all:
>
>  class CoClass(COMObject):
>      from comtypes._meta import _coclass_meta as __metaclass__

Uhm. Is that supposed to be a obfuscated trick for:

from comtypes._meta import _coclass_meta

class CoClass(COMObject):
      __metaclass__ = _coclass_meta

Because if it is I think it's better it's unobfoscated instead.

-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

From theller at ctypes.org  Mon Dec  8 21:54:35 2008
From: theller at ctypes.org (Thomas Heller)
Date: Mon, 08 Dec 2008 21:54:35 +0100
Subject: [Python-porting] Some questions about build_py_2to3
In-Reply-To: <319e029f0812081248i439b434dpa6c718a7d8b22449@mail.gmail.com>
References: <493D82FA.1080709@ctypes.org>
	<319e029f0812081248i439b434dpa6c718a7d8b22449@mail.gmail.com>
Message-ID: <493D898B.20409@ctypes.org>

[sorry, forgto to cc the list]

Lennart Regebro schrieb:
> > On Mon, Dec 8, 2008 at 21:26, Thomas Heller <theller at ctypes.org> wrote:
>> >> Here is one example that isn't converted at all:
>> >>
>> >>  class CoClass(COMObject):
>> >>      from comtypes._meta import _coclass_meta as __metaclass__
> > 
> > Uhm. Is that supposed to be a obfuscated trick for:

Yes it is.

> > 
> > from comtypes._meta import _coclass_meta
> > 
> > class CoClass(COMObject):
> >       __metaclass__ = _coclass_meta
> > 
> > Because if it is I think it's better it's unobfoscated instead.
> > 
This may be true.  Anyway, it is actual working code that the converter fails to convert.

-- Thanks, Thomas 

From martin at v.loewis.de  Mon Dec  8 23:25:53 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Dec 2008 23:25:53 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <079701c9593b$6c7b6b20$45724160$@com.au>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>
	<493D172F.6030304@egenix.com>
	<079701c9593b$6c7b6b20$45724160$@com.au>
Message-ID: <493D9EF1.3080303@v.loewis.de>

> It's more about moving to Python 2.6 syntax *only*.  eg, using 'bytes' or
> memory views etc as porting aids isn't that useful when targetting earlier
> versions. 

In the Django port, I do use (something like) bytes throughout the code,
and do so for all versions. I do

from django.utils.py3 import b

then

  b("Hallo")

which is is equivalent to b"Hallo", but works for all versions of
Python (it's implemented as .encode("ascii") in 3k)
(in addition, Django has its own SafeStr/SafeUnicode classes which
need special consideration)

I tend to ignore the suggestion of porting to 2.6 first, and also tend
to ignore the suggestion of using the -3 option. This is, of course,
because there are plenty of test cases for the software I'm porting,
so I can be reasonably certain that all aspects get covered. I then
don't need -3 to tell me what *might* break - I know for certain what
*did* break.

It would be an interesting experiment to test whether -3 still reports
anything after I completed porting - any such reports should then be
considered false positives.

Regards,
Martin

From martin at v.loewis.de  Tue Dec  9 00:18:32 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 09 Dec 2008 00:18:32 +0100
Subject: [Python-porting] Some questions about build_py_2to3
In-Reply-To: <493D82FA.1080709@ctypes.org>
References: <493D82FA.1080709@ctypes.org>
Message-ID: <493DAB48.2090700@v.loewis.de>

Thomas Heller wrote:
> I have some beginners questions about distutils build_py_2to3 command:
> 
> - Can it convert doctests?  How?

It only converts what lib2to3 converts - so the question really is
whether lib2to3 can convert doctests. AFAICT, no doctest fixer has been
written yet (although it should be possible to write on - probably on
an opt-in basis).

> - Shouldn't build_py_2to3 use a differnt build directory than the normal
> build_py command?  Perhaps build/lib-3 or something like that?

Perhaps. We would need to check first whether this is feasible to
implement (I don't know off-hand), as install_lib then needs to pick
up the right files. Then, the next question is whether changing the
layout risks compatibility with external tools, such as setuptools.

So for the time being, you have to clean the build directory when you
switch Python versions.

> - Should programming idioms that it fails to convert be reported as bugs?

On a case-by-case basis - yes, they should be reported, as 2to3 bugs.

> Here is one example that isn't converted at all:
> 
>   class CoClass(COMObject):
>       from comtypes._meta import _coclass_meta as __metaclass__
> 
> 
> It is not obvious immeditately how the converted code should look like,
> but at least the converter could issue a warning.

I don't think this can reasonably converted; you should rewrite it
first. This is typical - IMO, it is a true bug in 2to3 ONLY IF there
is no way to write it in 2.x so that 2to3 will convert it correctly.
If you can rewrite it by hand, then it is only a feature request for
2to3.

Channeling Guido (thereby moving to thin ice): It is a bug only if
python 2.6 -3 will not issue a warning, 2to3 does nothing, yet it
fails to run in 3.x. The bug then might be in -3 (for not issuing
a warning), or in 2to3 (for not converting), or in 3.x
(for rejecting).

In the specific case, it is easy to rewrite so that 2to3 can handle
it properly:

from comtypes._meta import _coclass_meta
class CoClass(COMObject):
   __metaclass__ = _coclass_meta
   ...
del _coclass_meta

Whether or not warnings should be issued is a different question.
Not sure whether 2to3 already issues warnings in some cases, and,
if it does, whether build_py_2to3 displays them.

Regards,
Martin

From musiccomposition at gmail.com  Tue Dec  9 02:47:24 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Mon, 8 Dec 2008 19:47:24 -0600
Subject: [Python-porting] Some questions about build_py_2to3
In-Reply-To: <493DAB48.2090700@v.loewis.de>
References: <493D82FA.1080709@ctypes.org> <493DAB48.2090700@v.loewis.de>
Message-ID: <1afaf6160812081747m4a16c224kb36966daf942a8a7@mail.gmail.com>

On Mon, Dec 8, 2008 at 5:18 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Thomas Heller wrote:
>> I have some beginners questions about distutils build_py_2to3 command:
>>
>> - Can it convert doctests?  How?
>
> It only converts what lib2to3 converts - so the question really is
> whether lib2to3 can convert doctests. AFAICT, no doctest fixer has been
> written yet (although it should be possible to write on - probably on
> an opt-in basis).

Yes, it can!

$ cat example.py
def add(a, b):
    """
    >>> print add(2, 4)
    6
    """
    return a + b

$ 2to3 -d example.py
--- example.py (original)
+++ example.py (refactored)
@@ -1,6 +1,6 @@
 def add(a, b):
     """
-    >>> print add(2, 4)
+    >>> print(add(2, 4))
     6
     """
     return a + b
RefactoringTool: Files that need to be modified:
RefactoringTool: example.py


> Whether or not warnings should be issued is a different question.
> Not sure whether 2to3 already issues warnings in some cases, and,
> if it does, whether build_py_2to3 displays them.

Warnings from 2to3 fixers get dumped into RefactoringTool's log_message.



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From mal at egenix.com  Tue Dec  9 13:23:22 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 09 Dec 2008 13:23:22 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <079701c9593b$6c7b6b20$45724160$@com.au>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>
	<493D172F.6030304@egenix.com>
	<079701c9593b$6c7b6b20$45724160$@com.au>
Message-ID: <493E633A.5020403@egenix.com>

On 2008-12-08 14:46, Mark Hammond wrote:
>>> Is there anyone else here for whom the step of "port to python 2.6"
>> is a deal-breaker?
>>
>> I'm not sure where the problems would lie in porting to Python 2.6
>> first.
> 
> It's more about moving to Python 2.6 syntax *only*.  eg, using 'bytes' or
> memory views etc as porting aids isn't that useful when targetting earlier
> versions.  Further, I meant "deal breaker" in a figurative sense - ie, not a
> real blocker, just an extra complication than what the "general" porting
> guide might bother considering - but by implication, something that might be
> worth addressing *somewhere*; eg, I'm finding '.encode('ascii')' a
> reasonable porting aid for bytes objects in many 'demo' byte literal cases,
> such as appending \0 chars, etc.
> 
> 'Cheers'.encode('ascii'), ly,

I think that one way or another you're going to end up using a
Python porting module that collects all the helpers you need to
support both Python 2 and 3.

The same is going to be needed for C extensions.

We're going to start moving all the Python version compatibility
stuff into a new module for our code base. For the C extensions
we already have a header file for the things we need in our
tools. With Python 3, we're likely going to have to complement this
with a separate C file for the various helper functions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 09 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2008-12-02: Released mxODBC.Connect 1.0.0      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From brett at python.org  Tue Dec  9 19:25:21 2008
From: brett at python.org (Brett Cannon)
Date: Tue, 9 Dec 2008 10:25:21 -0800
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <493E633A.5020403@egenix.com>
References: <493A5498.2070602@v.loewis.de>
	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>
	<493AFDF7.90006@python.org>
	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>
	<493C4412.9050004@gmail.com> <493D172F.6030304@egenix.com>
	<079701c9593b$6c7b6b20$45724160$@com.au> <493E633A.5020403@egenix.com>
Message-ID: <bbaeab100812091025o78fe195g5a490596692807d5@mail.gmail.com>

On Tue, Dec 9, 2008 at 04:23, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2008-12-08 14:46, Mark Hammond wrote:
>>>> Is there anyone else here for whom the step of "port to python 2.6"
>>> is a deal-breaker?
>>>
>>> I'm not sure where the problems would lie in porting to Python 2.6
>>> first.
>>
>> It's more about moving to Python 2.6 syntax *only*.  eg, using 'bytes' or
>> memory views etc as porting aids isn't that useful when targetting earlier
>> versions.  Further, I meant "deal breaker" in a figurative sense - ie, not a
>> real blocker, just an extra complication than what the "general" porting
>> guide might bother considering - but by implication, something that might be
>> worth addressing *somewhere*; eg, I'm finding '.encode('ascii')' a
>> reasonable porting aid for bytes objects in many 'demo' byte literal cases,
>> such as appending \0 chars, etc.
>>
>> 'Cheers'.encode('ascii'), ly,
>
> I think that one way or another you're going to end up using a
> Python porting module that collects all the helpers you need to
> support both Python 2 and 3.
>
> The same is going to be needed for C extensions.
>
> We're going to start moving all the Python version compatibility
> stuff into a new module for our code base. For the C extensions
> we already have a header file for the things we need in our
> tools. With Python 3, we're likely going to have to complement this
> with a separate C file for the various helper functions.
>

I suspect that, over time, best practices are going to emerge as
recipes that we should place in lib2to3 for people to use. Obviously
we should wait for those best practices to form, but definitely
something to be watching for.

-Brett

From sjmachin at lexicon.net  Thu Dec 11 23:15:36 2008
From: sjmachin at lexicon.net (John Machin)
Date: Fri, 12 Dec 2008 09:15:36 +1100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <079701c9593b$6c7b6b20$45724160$@com.au>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>
	<493D172F.6030304@egenix.com>
	<079701c9593b$6c7b6b20$45724160$@com.au>
Message-ID: <49419108.5010003@lexicon.net>

On 9/12/2008 00:46, Mark Hammond wrote:
>>> Is there anyone else here for whom the step of "port to python 2.6"
>> is a deal-breaker?
>>
>> I'm not sure where the problems would lie in porting to Python 2.6
>> first.
> 
> It's more about moving to Python 2.6 syntax *only*.  eg, using 'bytes' or
> memory views etc as porting aids isn't that useful when targetting earlier
> versions.  Further, I meant "deal breaker" in a figurative sense - ie, not a
> real blocker, just an extra complication than what the "general" porting
> guide might bother considering - but by implication, something that might be
> worth addressing *somewhere*; eg, I'm finding '.encode('ascii')' a
> reasonable porting aid for bytes objects in many 'demo' byte literal cases,
> such as appending \0 chars, etc.
> 

[already sent by mistake to Mark only]

Hello Mark [and others!],

I'm just starting to look at the ramifications of trying to support xlrd 
  on 2.1 to 2.6 plus 3.x with just one set of source files. It already 
has a timemachine module where all version-dependent code is isolated. 
Version-dependency is handled at load time by tests on sys.version. All 
other modules do:
     from timemachine import *
So far this has worked well enough with some compromises e.g. 
timemachine.py must be loadable on all supported versions, so this means 
all versions end up using "adict.has_key(k)" because "k in adict" causes 
a SyntaxError on 2.1.

It looks like I'm going to need to abandon the "one timemachine fits 
all" idea, and use customised ones: timemachine_21, ..._22_26, ..._3x. 
That leaves me with one distutils-related problem: it insists on 
compiling all included files when you do "setup.py install"; I don't 
want it to compile timemachine_21.py with Python 3.0  -- it would barf 
on "False = 0" :-) -- and vice versa. But I also don't want to have 
tarballs/installers that depend on the Python version, if possible.

One major annoyance so far is what to do with 2.x str literals. I'm 
resigned to going through all the string literals and ensuring that the 
"text" ones are written as u"foo" and that the others are written as 
....what??? I thought of making it b"foo" and lashing up a quick'n'dirty 
"3to2" back-porter.

Another possibility is writing it as b("foo"), with the timemachine 
defining an appropriate b function:
     b = lambda x: x # 2.x
     b = lambda x: x.encode('ascii') # 3.x
but the runtime overhead could be painful, especially in 3.x.

I do hope there's a simple solution that I've missed. What are others doing?

Cheers,
John





From martin at v.loewis.de  Fri Dec 12 00:01:17 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 12 Dec 2008 00:01:17 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <49419108.5010003@lexicon.net>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>	<493D172F.6030304@egenix.com>	<079701c9593b$6c7b6b20$45724160$@com.au>
	<49419108.5010003@lexicon.net>
Message-ID: <49419BBD.6000801@v.loewis.de>

> It looks like I'm going to need to abandon the "one timemachine fits
> all" idea, and use customised ones: timemachine_21, ..._22_26, ..._3x.
> That leaves me with one distutils-related problem: it insists on
> compiling all included files when you do "setup.py install"; I don't
> want it to compile timemachine_21.py with Python 3.0  -- it would barf
> on "False = 0" :-) -- and vice versa. But I also don't want to have
> tarballs/installers that depend on the Python version, if possible.

I recommend to use build_py_2to3. This will convert your 2.x source
files into 3.x syntax *at installation time*, and thus should solve
this problem.

> Another possibility is writing it as b("foo"), with the timemachine
> defining an appropriate b function:
>     b = lambda x: x # 2.x
>     b = lambda x: x.encode('ascii') # 3.x
> but the runtime overhead could be painful, especially in 3.x.

You could use interning if you are worried about the runtime overhead:

_intern = {}
def b(s):
  try:
    return _intern[s]
  except KeyError:
    res = _intern[s] = s.encode("ascii")
    return res

This will be still somewhat slower, but not much.

If you are really worried about performance, you could provide an
addition 2to3 fixer which unwraps the b() calls when converting to
3.x code. However, I'd rather make sure they aren't used in tight loops,
and just stop worrying.

Regards,
Martin

From mal at egenix.com  Fri Dec 12 14:24:22 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 12 Dec 2008 14:24:22 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <49419BBD.6000801@v.loewis.de>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>	<493D172F.6030304@egenix.com>	<079701c9593b$6c7b6b20$45724160$@com.au>	<49419108.5010003@lexicon.net>
	<49419BBD.6000801@v.loewis.de>
Message-ID: <49426606.704@egenix.com>

On 2008-12-12 00:01, Martin v. L?wis wrote:
>> It looks like I'm going to need to abandon the "one timemachine fits
>> all" idea, and use customised ones: timemachine_21, ..._22_26, ..._3x.
>> That leaves me with one distutils-related problem: it insists on
>> compiling all included files when you do "setup.py install"; I don't
>> want it to compile timemachine_21.py with Python 3.0  -- it would barf
>> on "False = 0" :-) -- and vice versa. But I also don't want to have
>> tarballs/installers that depend on the Python version, if possible.
> 
> I recommend to use build_py_2to3. This will convert your 2.x source
> files into 3.x syntax *at installation time*, and thus should solve
> this problem.

Actually, this happens at build time, but for source distributions
that's usually the same as installation time.

The problem with this approach is that 2to3 is a fast moving target
and relying on the version that comes with Python is likely not going
to work out if you need a more recent version of 2to3 for your code.

For source distributions this either means that you have to ship
2to3 with your code or that you ship already converted code.

BTW: Is it possible to extend 2to3 with your own fixers dynamically,
e.g. to work around this problem ?

Is there an overview of how lib2to3 works somewhere ?

>> Another possibility is writing it as b("foo"), with the timemachine
>> defining an appropriate b function:
>>     b = lambda x: x # 2.x
>>     b = lambda x: x.encode('ascii') # 3.x
>> but the runtime overhead could be painful, especially in 3.x.
> 
> You could use interning if you are worried about the runtime overhead:
> 
> _intern = {}
> def b(s):
>   try:
>     return _intern[s]
>   except KeyError:
>     res = _intern[s] = s.encode("ascii")
>     return res
> 
> This will be still somewhat slower, but not much.
> 
> If you are really worried about performance, you could provide an
> addition 2to3 fixer which unwraps the b() calls when converting to
> 3.x code. However, I'd rather make sure they aren't used in tight loops,
> and just stop worrying.

Adding a fixer is probably the better option. The fixer could then
convert b("abc") directly to b"abc" for Python 3.

That still leaves the function call for Python 2. Could 2to3 also
be used for Python 2 to 2 conversions ?

Looking at the code, there
doesn't appear to be anything specifically targeting Python 3 in
the fixer code, so I guess running "2to3 -f b_function_for_2 xyz.py"
could be used to convert b("abc") to "abc" for Python 2.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 12 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2008-12-02: Released mxODBC.Connect 1.0.0      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Sat Dec 13 00:13:15 2008
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 13 Dec 2008 00:13:15 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <49426606.704@egenix.com>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>	<493D172F.6030304@egenix.com>	<079701c9593b$6c7b6b20$45724160$@com.au>	<49419108.5010003@lexicon.net>
	<49419BBD.6000801@v.loewis.de> <49426606.704@egenix.com>
Message-ID: <4942F00B.90405@v.loewis.de>

> The problem with this approach is that 2to3 is a fast moving target
> and relying on the version that comes with Python is likely not going
> to work out if you need a more recent version of 2to3 for your code.

So program against the version that comes with Python. I found that this
works quite well.

Of course, it worked best when you reported all problems you have in
your software before 3.0 was released.

> For source distributions this either means that you have to ship
> 2to3 with your code or that you ship already converted code.
> 
> BTW: Is it possible to extend 2to3 with your own fixers dynamically,
> e.g. to work around this problem ?

Sure. You can pass explicit lists of fixers.

> Is there an overview of how lib2to3 works somewhere ?

How top-level do you want the overview to be? There is the fixes
directory, with one module per fixer. lib2to3 imports them all,
and then runs them one after another.

Regards,
Martin

From regebro at gmail.com  Sat Dec 13 12:45:02 2008
From: regebro at gmail.com (Lennart Regebro)
Date: Sat, 13 Dec 2008 12:45:02 +0100
Subject: [Python-porting] Porting Doctest exceptions
Message-ID: <319e029f0812130345u686a64a2n74a6ac0c9813be3b@mail.gmail.com>

In Python 2, an exception looks like this:

    Traceback (most recent call last):
      blahlalablablabl
    TheExceptionClass: Eh, something broke!

Python Python 3, an exception looks like this:

    Traceback (most recent call last):
      blahlalablablabl
    themodule.TheExceptionClass: Eh, something broke!

That extra "themodule." breaks every known doctest that checks for
exceptions in the universe, and I can't find any way of writing the
test that will work in both.

Any ideas of how to solve this?

-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

From Resul-Cetin at gmx.net  Sat Dec 13 22:21:12 2008
From: Resul-Cetin at gmx.net (Resul Cetin)
Date: Sat, 13 Dec 2008 22:21:12 +0100
Subject: [Python-porting] conversation of byte to str
Message-ID: <200812132221.12900.Resul-Cetin@gmx.net>

Hi,
I am currently porting a program which reads some xml data from https using 
urllib and reads it with minidom to create some other data for us. I expected
to get a str but all current data I get is bytes (which makes sense of course
as it is more common). So my idea was to do a bytes.decode("utf-8") but that
gives me a ascii error when it tries to decode utf-8 data (german umlauts in
this case).

What I do is:
 import urllib.request
 a = urllib.request.urlopen('http://www.example.com/')
 b = a.read()


b holds now
 b'<asd>\n\t<p>aa\xc3\xa4aa</p>\n</asd>\n'

When I try to decode that with
 str(b'<asd>\n\t<p>aa\xc3\xa4aa</p>\n</asd>\n', encoding="utf-8")
or
 b.decode("utf-8")

I get
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python3.0/io.py", line 1491, in write
     b = encoder.encode(s)
   File "/usr/lib/python3.0/encodings/ascii.py", line 22, in encode
     return codecs.ascii_encode(input, self.errors)[0]
 UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 
15: ordinal 
not in range(128)


b.decode("utf-8", "ignore") work either... I am extrem glueless how to get a 
string to 
feed minidom with it.

PS: I got my version from ubuntu jaunty today

Regards,
	Resul Cetin



From ivazqueznet at gmail.com  Sat Dec 13 22:35:33 2008
From: ivazqueznet at gmail.com (Ignacio Vazquez-Abrams)
Date: Sat, 13 Dec 2008 16:35:33 -0500
Subject: [Python-porting] conversation of byte to str
In-Reply-To: <200812132221.12900.Resul-Cetin@gmx.net>
References: <200812132221.12900.Resul-Cetin@gmx.net>
Message-ID: <1229204133.30148.279.camel@ignacio.lan>

On Sat, 2008-12-13 at 22:21 +0100, Resul Cetin wrote:
> b holds now
>  b'<asd>\n\t<p>aa\xc3\xa4aa</p>\n</asd>\n'
> 
> When I try to decode that with
>  str(b'<asd>\n\t<p>aa\xc3\xa4aa</p>\n</asd>\n', encoding="utf-8")
> or
>  b.decode("utf-8")
> 
> I get
>  Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/usr/lib/python3.0/io.py", line 1491, in write
>      b = encoder.encode(s)
>    File "/usr/lib/python3.0/encodings/ascii.py", line 22, in encode
>      return codecs.ascii_encode(input, self.errors)[0]
>  UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 
> 15: ordinal 
> not in range(128)

No, the decoding is working properly. The problem is elsewhere, probably
when you try to print the result.

-- 
Ignacio Vazquez-Abrams <ivazqueznet at gmail.com>

PLEASE don't CC me; I'm already subscribed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-porting/attachments/20081213/4a5ac5f1/attachment.pgp>

From Resul-Cetin at gmx.net  Sat Dec 13 23:31:42 2008
From: Resul-Cetin at gmx.net (Resul Cetin)
Date: Sat, 13 Dec 2008 23:31:42 +0100
Subject: [Python-porting] conversation of byte to str
In-Reply-To: <1229204133.30148.279.camel@ignacio.lan>
References: <200812132221.12900.Resul-Cetin@gmx.net>
	<1229204133.30148.279.camel@ignacio.lan>
Message-ID: <200812132331.42835.Resul-Cetin@gmx.net>

On Saturday 13 December 2008 22:35:33 Ignacio Vazquez-Abrams wrote:
> No, the decoding is working properly. The problem is elsewhere, probably
> when you try to print the result.
Thx, You are correct. Sometimes I make more errors while searching
for them :D
The correct problem seems to be in urllib. 
Try for example (exchange USERNAME with your delicious username and PASSWORD with your 
delicious password):
 import urllib.request
 auth_handler = urllib.request.HTTPBasicAuthHandler()
 auth_handler.add_password('del.icio.us API', 'api.del.icio.us', USERNAME, PASSWORD)
 opener = urllib.request.build_opener(auth_handler)
 print(str(opener.open('https://api.del.icio.us/v1/posts/all').read(20), "utf-8"))

And you will see some bogus extra data at the beginning I dont have with
curl, wget or python 2.5.

Regards,
	Resul

From Resul-Cetin at gmx.net  Sun Dec 14 11:22:28 2008
From: Resul-Cetin at gmx.net (Resul Cetin)
Date: Sun, 14 Dec 2008 11:22:28 +0100
Subject: [Python-porting] conversation of byte to str
In-Reply-To: <200812132331.42835.Resul-Cetin@gmx.net>
References: <200812132221.12900.Resul-Cetin@gmx.net>
	<1229204133.30148.279.camel@ignacio.lan>
	<200812132331.42835.Resul-Cetin@gmx.net>
Message-ID: <200812141122.28511.Resul-Cetin@gmx.net>

On Saturday 13 December 2008 23:31:42 Resul Cetin wrote:
> On Saturday 13 December 2008 22:35:33 Ignacio Vazquez-Abrams wrote:
> > No, the decoding is working properly. The problem is elsewhere, probably
> > when you try to print the result.
>
> Thx, You are correct. Sometimes I make more errors while searching
> for them :D
> The correct problem seems to be in urllib.
> Try for example (exchange USERNAME with your delicious username and
> PASSWORD with your delicious password):
>  import urllib.request
>  auth_handler = urllib.request.HTTPBasicAuthHandler()
>  auth_handler.add_password('del.icio.us API', 'api.del.icio.us', USERNAME,
> PASSWORD) opener = urllib.request.build_opener(auth_handler)
>  print(str(opener.open('https://api.del.icio.us/v1/posts/all').read(20),
> "utf-8"))
>
> And you will see some bogus extra data at the beginning I dont have with
> curl, wget or python 2.5.

It seems to be related with that bug: http://bugs.python.org/issue4631


From sjmachin at lexicon.net  Mon Dec 15 12:03:35 2008
From: sjmachin at lexicon.net (John Machin)
Date: Mon, 15 Dec 2008 22:03:35 +1100
Subject: [Python-porting] bytes != str ... a few notes
Message-ID: <49463987.6010408@lexicon.net>

In Python 3, b'abc' != 'abc' (and rightly so). I have scribbled down 
some notes (below) which may be useful to others. Is there a "porting 
tips" wiki or other public place for posting this kind of thing?

A couple of minor points on other topics:

1. It would have been nice if 3.x ord(a_byte) just quietly returned 
a_byte; porters would have not needed to change anything.

2. bytes.join() and bytearray.join() exist and work (on an iterable 
which may contain a mixture of bytes and bytearray objects) just as 
extrapolation from str.join would lead you to expect, but the help needs 
a little fixing and there's no mention of them in the Library Reference 
Manual. I've raised a bug report: http://bugs.python.org/issue4669

Cheers,
John

=== Comparing bytes objects with str objects ===

In Python 3.x, a bytes object will never ever compare equal to a str object.

Porter's problem (example):

data = open(fpath, "rb").read(8)
OLE2_SIG = "\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1"
if data == OLE2_SIG:
     # This point is unreachable in 3.x, because data is bytes (has been
     # read from a file opened in binary mode) and OLE_2SIG is a str
     # object.

Solution for "simple" porting:
OLE2_SIG = b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1"

A tentative solution when maintaining one codebase which runs as is on 
2.x and from which 3.x code is generated:

# ... excerpt from "include file"
if python_version >= (3, 0):
     def STR2BYTES(x, encoding='latin1'):
         return x.encode(encoding)
else:
     def STR2BYTES(x):
         return x

# ... changed code
OLE2_SIG = STR2BYTES("\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1")

How to find cases of this problem:
1. Can't be detected by Python 2.6 -3 option.
2. Can't be detected/handled by 2to3 script.
3. Is detected by Python 3.x -b (warn) and -bb (error) command-line 
options to check for bytes/str [in]equality comparisons. [Aside: these 
options are documented in the expected place but not mentioned in the 
porting notes 
(http://docs.python.org/dev/py3k/whatsnew/3.0.html#porting-to-python-3-0)]
4. Should be detected by your tests but the point where the test fails 
may be some distance from the actual comparison.
5. Search your code for bytesy things like \x and \0.
6. Read your code (but turn 2.x mindset off because if you don't, the 
code will look just fine!).

=== end of screed ===

From python at mikewatkins.ca  Mon Dec 15 19:01:50 2008
From: python at mikewatkins.ca (Michael Watkins)
Date: Mon, 15 Dec 2008 10:01:50 -0800 (PST)
Subject: [Python-porting] bytes != str ... a few notes
Message-ID: <3407.24.84.177.146.1229364110.squirrel@webmail.solutionroute.ca>

On Mon, December 15, 2008 3:03 am, John Machin wrote:
> === Comparing bytes objects with str objects ===
> A tentative solution when maintaining one codebase which runs as is on
> 2.x and from which 3.x code is generated:
>
> if python_version >= (3, 0):
>     def STR2BYTES(x, encoding='latin1'):
>         return x.encode(encoding)
> else:
>     def STR2BYTES(x): return x

Perhaps your STR2BYTES function should test to see if "x" is already a
byte string, to avoid recasting errors. As it stands should "x" be recast
later down the road by some other chunk of code which is oblivious to the
history of the recast "x", it will fail with an AttributeError:

>>> STR2BYTES(something_already_a_byte_string)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in STR2BYTES
AttributeError: 'bytes' object has no attribute 'encode'

An approach similar to yours is what the authors of Durus, a ZODB-like
Python Object Database, have done. They add an isinstance(s, byte_string)
test to avoid any attempt at re-encoding a byte string (which would lead
to an attribute error since a byte string will never have an "encode"
method.

Sadly the same is not true in 2.x and below.

Browse the relevant module:

http://www.mems-exchange.org/software/durus/Durus-3.8.tar.gz/Durus-3.8/utils.py

Or peek at this snippet from within::

    if sys.version < "3":
        from __builtin__ import xrange
        from __builtin__ import str as byte_string
        def iteritems(x):
            return x.iteritems()
        def next(x):
            return x.next()
        from cStringIO import StringIO as BytesIO
        from cPickle import dumps, loads, Unpickler, Pickler
    else:
        xrange = range
        from builtins import next, bytearray, bytes
        byte_string = (bytearray, bytes)
        def iteritems(x):
            return x.items()
        from io import BytesIO
        from pickle import dumps, loads, Unpickler, Pickler

    def as_bytes(s):
        """Return a byte_string produced from the string s."""
        if isinstance(s, byte_string):
            return s
        else:
            return s.encode('latin1')

    empty_byte_string = as_bytes("")

I wish it were as easy as searching for '\x'-y looking literals to find
areas that will work in 2 but fail in 3. That's a start but there are
little surprises to find elsewhere. Consider the following which of course
won't fail on Python < 3 but will on >= 3::

    Python 3:
    >>> x = ":".join(('1','plus','two'))
    >>> x
    '1:plus:two'
    >>> hashlib.md5(x)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: object supporting the buffer API required

    # an easy cross platform fix:
    >>> hashlib.md5(as_bytes(x))
    <md5 HASH object @ 0x8933400>
    >>> hashlib.md5(as_bytes(x)).hexdigest()
    '4e3a3a8075a6982177c24af5179ec82c'

Failing code and failing unit tests ought to pick up most of these sorts
of issues.




From mal at egenix.com  Mon Dec 15 20:55:49 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 15 Dec 2008 20:55:49 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <4942F00B.90405@v.loewis.de>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>	<493D172F.6030304@egenix.com>	<079701c9593b$6c7b6b20$45724160$@com.au>	<49419108.5010003@lexicon.net>	<49419BBD.6000801@v.loewis.de>
	<49426606.704@egenix.com> <4942F00B.90405@v.loewis.de>
Message-ID: <4946B645.4070903@egenix.com>

On 2008-12-13 00:13, Martin v. L?wis wrote:
>> The problem with this approach is that 2to3 is a fast moving target
>> and relying on the version that comes with Python is likely not going
>> to work out if you need a more recent version of 2to3 for your code.
> 
> So program against the version that comes with Python. I found that this
> works quite well.
>
> Of course, it worked best when you reported all problems you have in
> your software before 3.0 was released.

In an ideal world that would have happened, but in the real world
Python 2.x development is still what pays the bill.

>> For source distributions this either means that you have to ship
>> 2to3 with your code or that you ship already converted code.
>>
>> BTW: Is it possible to extend 2to3 with your own fixers dynamically,
>> e.g. to work around this problem ?
> 
> Sure. You can pass explicit lists of fixers.

Can I register new ones somewhere ? (other than copying them into the
fixes package dir or monkey patching them in there)

>> Is there an overview of how lib2to3 works somewhere ?
> 
> How top-level do you want the overview to be? There is the fixes
> directory, with one module per fixer. lib2to3 imports them all,
> and then runs them one after another.

Well, I know about the http://docs.python.org/library/2to3.htm page
and can read the source code, but I'm missing an overview of the
fixers API and the pattern definition language used in them.

The API looks rather complicated, even for very simple search and
replace tasks such as __nonzero__ -> __bool__ conversion, e.g. from
fix_nonzero.py:

class FixNonzero(fixer_base.BaseFix):
    PATTERN = """
    classdef< 'class' any+ ':'
              suite< any*
                     funcdef< 'def' name='__nonzero__'
                              parameters< '(' NAME ')' > any+ >
                     any* > >
    """

    def transform(self, node, results):
        name = results["name"]
        new = Name("__bool__", prefix=name.get_prefix())
        name.replace(new)

PATTERN looks like some kind of tree pattern recognition language.
I would assume that node is the found node, but have no idea
what results means. prefix also is a mystery - method names don't
have prefixes, so this has got to be something else.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 15 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2008-12-02: Released mxODBC.Connect 1.0.0      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From martin at v.loewis.de  Mon Dec 15 22:17:47 2008
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 15 Dec 2008 22:17:47 +0100
Subject: [Python-porting] Port of psycopg2
In-Reply-To: <4946B645.4070903@egenix.com>
References: <493A5498.2070602@v.loewis.de>	<bbaeab100812061412l1f325696mc53e618e5492b86d@mail.gmail.com>	<493AFDF7.90006@python.org>	<bbaeab100812061609o708a7ccfp3c895b39176766b3@mail.gmail.com>	<493C4412.9050004@gmail.com>	<493D172F.6030304@egenix.com>	<079701c9593b$6c7b6b20$45724160$@com.au>	<49419108.5010003@lexicon.net>	<49419BBD.6000801@v.loewis.de>
	<49426606.704@egenix.com> <4942F00B.90405@v.loewis.de>
	<4946B645.4070903@egenix.com>
Message-ID: <4946C97B.9080206@v.loewis.de>

> Can I register new ones somewhere ? (other than copying them into the
> fixes package dir or monkey patching them in there)

No need to register them. Just pass the module names.

> The API looks rather complicated, even for very simple search and
> replace tasks such as __nonzero__ -> __bool__ conversion, e.g. from
> fix_nonzero.py:
> 
> class FixNonzero(fixer_base.BaseFix):
>     PATTERN = """
>     classdef< 'class' any+ ':'
>               suite< any*
>                      funcdef< 'def' name='__nonzero__'
>                               parameters< '(' NAME ')' > any+ >
>                      any* > >
>     """
> 
>     def transform(self, node, results):
>         name = results["name"]
>         new = Name("__bool__", prefix=name.get_prefix())
>         name.replace(new)
> 
> PATTERN looks like some kind of tree pattern recognition language.
> I would assume that node is the found node, but have no idea
> what results means. prefix also is a mystery - method names don't
> have prefixes, so this has got to be something else.

You shouldn't try to write the patterns yourself, but instead use
the find_pattern.py tool from the 2to3 sandbox. The node that
matches follows Grammar/Grammar, where classdef is defined as

classdef: 'class' NAME ['(' [testlist] ')'] ':' suite

and the embedded productions are

suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT
funcdef: 'def' NAME parameters ':' suite
parameters: '(' [varargslist] ')'

prefix is the lexical prefix of the token, i.e. typically the
set of whitespace preceding the name. This is necessary
so that __bool__ gets into the same column that __nonzero__
was in, in case you wrote

   def           __nonzero__(self):pass

Regards,
Martin


From csad7 at t-online.de  Mon Dec 15 21:18:41 2008
From: csad7 at t-online.de (see)
Date: Mon, 15 Dec 2008 21:18:41 +0100
Subject: [Python-porting] HTMLParser in 2to3
Message-ID: <4946BBA1.1010707@gmail.com>

hi,
(hope this is the right forum for this kind of question...)

While testing the 2to3 tool for a lib of mine I noticed imports for 
HTMLParser seem not to be changed yet.

I guess the following should happen:

	import HTMLParser
	# or
	from HTMLParser import HTMLParser

should get
	
	import html.parser
	# or
	from html.parser import HTMLParser

?

thanks
Chris

From brett at python.org  Mon Dec 15 23:44:09 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 15 Dec 2008 14:44:09 -0800
Subject: [Python-porting] HTMLParser in 2to3
In-Reply-To: <4946BBA1.1010707@gmail.com>
References: <4946BBA1.1010707@gmail.com>
Message-ID: <bbaeab100812151444g21437119k3e3ed993f06c6cba@mail.gmail.com>

On Mon, Dec 15, 2008 at 12:18, see <csad7 at t-online.de> wrote:
> hi,
> (hope this is the right forum for this kind of question...)
>
> While testing the 2to3 tool for a lib of mine I noticed imports for
> HTMLParser seem not to be changed yet.
>

Please file a bug report at http://bugs.python.org so this doesn't get lost.

> I guess the following should happen:
>
>        import HTMLParser
>        # or
>        from HTMLParser import HTMLParser
>
> should get
>
>        import html.parser
>        # or
>        from html.parser import HTMLParser

The last should be ``from html import pasrser as HTMLParser``.

-Brett

From musiccomposition at gmail.com  Tue Dec 16 03:10:06 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Mon, 15 Dec 2008 20:10:06 -0600
Subject: [Python-porting] HTMLParser in 2to3
In-Reply-To: <bbaeab100812151444g21437119k3e3ed993f06c6cba@mail.gmail.com>
References: <4946BBA1.1010707@gmail.com>
	<bbaeab100812151444g21437119k3e3ed993f06c6cba@mail.gmail.com>
Message-ID: <1afaf6160812151810s41977d8dg44084e728f0a21de@mail.gmail.com>

On Mon, Dec 15, 2008 at 4:44 PM, Brett Cannon <brett at python.org> wrote:
> On Mon, Dec 15, 2008 at 12:18, see <csad7 at t-online.de> wrote:
>> hi,
>> (hope this is the right forum for this kind of question...)
>>
>> While testing the 2to3 tool for a lib of mine I noticed imports for
>> HTMLParser seem not to be changed yet.
>>
>
> Please file a bug report at http://bugs.python.org so this doesn't get lost.

Don't bother filing a report; this is fixed in the 2to3 trunk:

$ ./2to3 -
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
import HTMLParser
from HTMLParser import HTMLParser
--- <stdin> (original)
+++ <stdin> (refactored)
@@ -1,2 +1,2 @@
-import HTMLParser
-from HTMLParser import HTMLParser
+import html.parser
+from html.parser import HTMLParser
RefactoringTool: Files that need to be modified:
RefactoringTool: <stdin>





-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From csad7 at t-online.de  Tue Dec 16 20:35:11 2008
From: csad7 at t-online.de (see)
Date: Tue, 16 Dec 2008 20:35:11 +0100
Subject: [Python-porting] HTMLParser in 2to3
In-Reply-To: <1afaf6160812151810s41977d8dg44084e728f0a21de@mail.gmail.com>
References: <4946BBA1.1010707@gmail.com>	<bbaeab100812151444g21437119k3e3ed993f06c6cba@mail.gmail.com>
	<1afaf6160812151810s41977d8dg44084e728f0a21de@mail.gmail.com>
Message-ID: <494802EF.2040201@gmail.com>

Benjamin Peterson wrote:
> On Mon, Dec 15, 2008 at 4:44 PM, Brett Cannon <brett at python.org> wrote:
>   
>> On Mon, Dec 15, 2008 at 12:18, see <csad7 at t-online.de> wrote:
>>     
>>> hi,
>>> (hope this is the right forum for this kind of question...)
>>>
>>> While testing the 2to3 tool for a lib of mine I noticed imports for
>>> HTMLParser seem not to be changed yet.
>>>
>>>       
>> Please file a bug report at http://bugs.python.org so this doesn't get lost.
>>     
>
> Don't bother filing a report; this is fixed in the 2to3 trunk:
>
> $ ./2to3 -
> RefactoringTool: Skipping implicit fixer: buffer
> RefactoringTool: Skipping implicit fixer: idioms
> RefactoringTool: Skipping implicit fixer: set_literal
> RefactoringTool: Skipping implicit fixer: ws_comma
> import HTMLParser
> from HTMLParser import HTMLParser
> --- <stdin> (original)
> +++ <stdin> (refactored)
> @@ -1,2 +1,2 @@
> -import HTMLParser
> -from HTMLParser import HTMLParser
> +import html.parser
> +from html.parser import HTMLParser
> RefactoringTool: Files that need to be modified:
> RefactoringTool: <stdin>
>   
great, thanks.

BTW, will the 2to3 tool be released in other cycles than Py3k, guess not 
every small bugfix will trigger a full release...
So I guess my question would be, is there a separate download somewhere?

Chris


From csad7 at t-online.de  Tue Dec 16 21:58:33 2008
From: csad7 at t-online.de (Christof Hoeke)
Date: Tue, 16 Dec 2008 21:58:33 +0100
Subject: [Python-porting]  HTMLParser in 2to3
Message-ID: <49481679.6010202@t-online.de>

 >> I guess the following should happen:
 >>
 >>        import HTMLParser
 >>        # or
 >>        from HTMLParser import HTMLParser
 >>
 >> should get
 >>
 >>        import html.parser
 >>        # or
 >>        from html.parser import HTMLParser

 > The last should be ``from html import pasrser as HTMLParser``.
 >
 > -Brett

not to want to split hairs but the second example ``from HTMLParser 
import HTMLParser`` actually imports the HTMLParser class and not the 
module so I think my original thinking was right?

thanks
Chris

From fdrake at acm.org  Wed Dec 17 03:06:17 2008
From: fdrake at acm.org (Fred Drake)
Date: Tue, 16 Dec 2008 21:06:17 -0500
Subject: [Python-porting] HTMLParser in 2to3
In-Reply-To: <49481679.6010202@t-online.de>
References: <49481679.6010202@t-online.de>
Message-ID: <CA52E5AB-E62D-4CF4-A218-D1D68D535F8B@acm.org>

On Dec 16, 2008, at 3:58 PM, Christof Hoeke wrote:
> not to want to split hairs but the second example ``from HTMLParser  
> import HTMLParser`` actually imports the HTMLParser class and not  
> the module so I think my original thinking was right?


Please, split hairs!

Yes, your original proposed re-writes are correct.


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From musiccomposition at gmail.com  Wed Dec 17 04:14:19 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Tue, 16 Dec 2008 21:14:19 -0600
Subject: [Python-porting] HTMLParser in 2to3
In-Reply-To: <494802EF.2040201@gmail.com>
References: <4946BBA1.1010707@gmail.com>
	<bbaeab100812151444g21437119k3e3ed993f06c6cba@mail.gmail.com>
	<1afaf6160812151810s41977d8dg44084e728f0a21de@mail.gmail.com>
	<494802EF.2040201@gmail.com>
Message-ID: <1afaf6160812161914j6eec65d5w63d4ef59c4c3225e@mail.gmail.com>

On Tue, Dec 16, 2008 at 1:35 PM, see <csad7 at t-online.de> wrote:
> great, thanks.
>
> BTW, will the 2to3 tool be released in other cycles than Py3k, guess not
> every small bugfix will trigger a full release...
> So I guess my question would be, is there a separate download somewhere?

You can download the current version from SVN. (It's kept quite stable.)

http://svn.python.org/projects/sandbox/trunk/2to3
>
> Chris
>
>



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From martin at v.loewis.de  Wed Dec 17 07:16:37 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 17 Dec 2008 07:16:37 +0100
Subject: [Python-porting] HTMLParser in 2to3
In-Reply-To: <494802EF.2040201@gmail.com>
References: <4946BBA1.1010707@gmail.com>	<bbaeab100812151444g21437119k3e3ed993f06c6cba@mail.gmail.com>	<1afaf6160812151810s41977d8dg44084e728f0a21de@mail.gmail.com>
	<494802EF.2040201@gmail.com>
Message-ID: <49489945.7020204@v.loewis.de>

> BTW, will the 2to3 tool be released in other cycles than Py3k, guess not
> every small bugfix will trigger a full release...
> So I guess my question would be, is there a separate download somewhere?

So far, there has not been a single stand-alone release of 2to3. People
wanting to use a newer version have always checked from subversion.

Regards,
Martin

From xi at gamma.dn.ua  Tue Dec 30 23:03:16 2008
From: xi at gamma.dn.ua (Kirill Simonov)
Date: Wed, 31 Dec 2008 00:03:16 +0200
Subject: [Python-porting] [ANN] PyYAML-3.08: Now with Python 3 support
Message-ID: <495A9AA4.4050406@gamma.dn.ua>

========================
  Announcing PyYAML-3.08
========================

A new release of PyYAML is now available:

     http://pyyaml.org/wiki/PyYAML

This release features complete support for Python 3.  For
compatibility notes between Python 2 and Python 3 versions,
please see

     http://pyyaml.org/wiki/PyYAMLDocumentation#Python3support


Changes
=======

* Python 3 support (Thank to Erick Tryzelaar).
* Use Cython instead of Pyrex to build LibYAML bindings.  Note
   that the source package is distributed with a pre-generated
   '_yaml.c' file so you don't need Cython installed to build
   LibYAML bindings.
* Refactored support for unicode and byte input/output streams.


Resources
=========

PyYAML homepage: http://pyyaml.org/wiki/PyYAML
PyYAML documentation: http://pyyaml.org/wiki/PyYAMLDocumentation

TAR.GZ package: http://pyyaml.org/download/pyyaml/PyYAML-3.08.tar.gz
ZIP package: http://pyyaml.org/download/pyyaml/PyYAML-3.08.zip
Windows installers:
     http://pyyaml.org/download/pyyaml/PyYAML-3.08.win32-py2.3.exe
     http://pyyaml.org/download/pyyaml/PyYAML-3.08.win32-py2.4.exe
     http://pyyaml.org/download/pyyaml/PyYAML-3.08.win32-py2.5.exe
     http://pyyaml.org/download/pyyaml/PyYAML-3.08.win32-py2.6.exe
     http://pyyaml.org/download/pyyaml/PyYAML-3.08.win32-py3.0.exe

PyYAML SVN repository: http://svn.pyyaml.org/pyyaml
Submit a bug report: http://pyyaml.org/newticket?component=pyyaml

YAML homepage: http://yaml.org/
YAML-core mailing list: 
http://lists.sourceforge.net/lists/listinfo/yaml-core


About PyYAML
============

YAML is a data serialization format designed for human readability and
interaction with scripting languages.  PyYAML is a YAML parser and
emitter for Python.

PyYAML features a complete YAML 1.1 parser, Unicode support, pickle
support, capable extension API, and sensible error messages.  PyYAML
supports standard YAML tags and provides Python-specific tags that allow
to represent an arbitrary Python object.

PyYAML is applicable for a broad range of tasks from complex
configuration files to object serialization and persistance.


Example
=======

 >>> import yaml

 >>> yaml.load("""
... name: PyYAML
... description: YAML parser and emitter for Python
... homepage: http://pyyaml.org/wiki/PyYAML
... keywords: [YAML, serialization, configuration, persistance, pickle]
... """)
{'keywords': ['YAML', 'serialization', 'configuration', 'persistance',
'pickle'], 'homepage': 'http://pyyaml.org/wiki/PyYAML', 'description':
'YAML parser and emitter for Python', 'name': 'PyYAML'}

 >>> print yaml.dump(_)
name: PyYAML
homepage: http://pyyaml.org/wiki/PyYAML
description: YAML parser and emitter for Python
keywords: [YAML, serialization, configuration, persistance, pickle]


Copyright
=========

The PyYAML module is written by Kirill Simonov <xi at resolvent.net>.

PyYAML is released under the MIT license.