From benjamin at python.org  Wed Nov 23 06:45:25 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 22 Nov 2011 23:45:25 -0600
Subject: [Python-porting] [RELEASED] six 1.1
Message-ID: <CAPZV6o_zapLFbEw41mhqymo2t1dQ17Jd9Uj_9kp22ft+jm88sg@mail.gmail.com>

I'm pleased to announce the immediate availability of six 1.1.0. six
is a small compatibility library for writing code that works on Python
2 and 3 without modification.

six 1.1 features several incremental improvements over 1.0. The
complete list of changes is:

- Add the int2byte function for converting an int of value less than
256 to a bytes object.

- Add compatibility mappings for iterators over the keys, values, and
items of a dictionary.

- Fix six.MAXSIZE on platforms where sizeof(long) != sizeof(Py_ssize_t).

- Issue #3: Add six.moves mappings for filter, map, and zip.


You can download six on PyPi:
    http://pypi.python.org/pypi/six

The documentation is at:
    http://packages.python.org/six/

Please report bugs at:
    http://bitbucket.org/gutworth/six


Regards,
Benjamin

From barry at python.org  Wed Nov 23 16:14:55 2011
From: barry at python.org (Barry Warsaw)
Date: Wed, 23 Nov 2011 10:14:55 -0500
Subject: [Python-porting] [RELEASED] six 1.1
In-Reply-To: <CAPZV6o_zapLFbEw41mhqymo2t1dQ17Jd9Uj_9kp22ft+jm88sg@mail.gmail.com>
References: <CAPZV6o_zapLFbEw41mhqymo2t1dQ17Jd9Uj_9kp22ft+jm88sg@mail.gmail.com>
Message-ID: <20111123101455.4d12402f@limelight.wooz.org>

On Nov 22, 2011, at 11:45 PM, Benjamin Peterson wrote:

>I'm pleased to announce the immediate availability of six 1.1.0. six
>is a small compatibility library for writing code that works on Python
>2 and 3 without modification.

Nice.  I skimmed the documentation, and there are definitely a few things that
I think would have been helpful in my current porting tasks.  I especially
like with_metaclass() since that's the one syntax that I've had to resort to
exec'ing for.

There are some good porting guides out there, but I've found some holes, at
both the Python and C levels.  I'll write up the details (e.g. __next__()
vs. next()) hopefully today, I've also found a few more traps and tricks for
extension modules.  I wonder if you have any interest in adding some C level
portability helpers.

Cheers,
-Barry

From benjamin at python.org  Wed Nov 23 17:55:42 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 23 Nov 2011 10:55:42 -0600
Subject: [Python-porting] [RELEASED] six 1.1
In-Reply-To: <20111123101455.4d12402f@limelight.wooz.org>
References: <CAPZV6o_zapLFbEw41mhqymo2t1dQ17Jd9Uj_9kp22ft+jm88sg@mail.gmail.com>
	<20111123101455.4d12402f@limelight.wooz.org>
Message-ID: <CAPZV6o9d72v3gfL+qtngiRnx+KjNhTuPufEjcd+k4=+3wWNrdw@mail.gmail.com>

2011/11/23 Barry Warsaw <barry at python.org>:
> On Nov 22, 2011, at 11:45 PM, Benjamin Peterson wrote:
>
>>I'm pleased to announce the immediate availability of six 1.1.0. six
>>is a small compatibility library for writing code that works on Python
>>2 and 3 without modification.
> Nice. ?I skimmed the documentation, and there are definitely a few things that
> I think would have been helpful in my current porting tasks. ?I especially
> like with_metaclass() since that's the one syntax that I've had to resort to
> exec'ing for.

Glad to be helpful. :)

>
> There are some good porting guides out there, but I've found some holes, at
> both the Python and C levels. ?I'll write up the details (e.g. __next__()
> vs. next()) hopefully today, I've also found a few more traps and tricks for
> extension modules. ?I wonder if you have any interest in adding some C level
> portability helpers.

You mean like a header file with macros for PyInt -> PyLong/PyString
-> PyUnicode etc?


-- 
Regards,
Benjamin

From barry at python.org  Wed Nov 23 21:59:15 2011
From: barry at python.org (Barry Warsaw)
Date: Wed, 23 Nov 2011 15:59:15 -0500
Subject: [Python-porting] [RELEASED] six 1.1
In-Reply-To: <CAPZV6o9d72v3gfL+qtngiRnx+KjNhTuPufEjcd+k4=+3wWNrdw@mail.gmail.com>
References: <CAPZV6o_zapLFbEw41mhqymo2t1dQ17Jd9Uj_9kp22ft+jm88sg@mail.gmail.com>
	<20111123101455.4d12402f@limelight.wooz.org>
	<CAPZV6o9d72v3gfL+qtngiRnx+KjNhTuPufEjcd+k4=+3wWNrdw@mail.gmail.com>
Message-ID: <20111123155915.2fd97971@limelight.wooz.org>

On Nov 23, 2011, at 10:55 AM, Benjamin Peterson wrote:

>2011/11/23 Barry Warsaw <barry at python.org>:
>> both the Python and C levels. ?I'll write up the details (e.g. __next__()
>> vs. next()) hopefully today, I've also found a few more traps and tricks for
>> extension modules. ?I wonder if you have any interest in adding some C level
>> portability helpers.
>
>You mean like a header file with macros for PyInt -> PyLong/PyString
>-> PyUnicode etc?

There are a bunch of little things I've found helpful while porting
dbus-python.  I think some at least would be generally useful for extension
modules.  Here's a quick summary (so far :).  I should note first that I only
care about Python 2.6, 2.7, and 3.2.  I think there was only one case where
2.6 didn't have what I needed.

I tried to reduce the number of #ifdefs in the code by converting some things
that can be made common between the two versions.

 - In Python 2, I always #include <bytesobject.h> and unilaterally change all
   PyString names to PyBytes names.  That reduces a lot of the ugliness.

 - I changed all the reprs to return unicodes in both Python versions instead
   of conditionally continuing to return strings in Python 2.  That reduced
   another source of noise, but I had to use a little trick with
   PyUnicode_FromFormat().  The reprs in this package embed the repr of the
   parent class, but you don't know whether that will be a bytes (under Python
   2) or a unicode (under Python 3).  It was fairly ugly to ifdef around this,
   so instead of using either the %s or %U codes wrapped in macros, I use the
   %V code.  Now, I'm not sure if that was added for this purpose, but it sure
   is handy.  The call sites look something like this now:

   PyObject *parent_repr = (<baseclass>.tp_repr)(self);
   PyObject *my_repr = PyUnicode_FromFormat("...%V...", REPRV(parent_repr));

   and the macro looks like this:

   #define REPRV(obj) \
       (PyUnicode_Check(obj) ? (obj) : NULL), \
       (PyUnicode_Check(obj) ? NULL : PyBytes_AS_STRING(obj))

   I supposed technically this could crash if the parent repr (erroneously)
   returned a non-string, but in my case, that won't happen because the base
   classes are standard Python types, or otherwise well-controlled.

Additional compatibility macros and functions:

 - I really dislike writing "#if PY_MAJOR_VERSION >= 3" all over the place, so
   I define the following macro to make the version test easier:

   #if PY_MAJOR_VERSION >= 3
   #define PY3K
   #endif

   Now all I need are "#ifdef PY3K" sprinkles.  Okay, maybe it's a minor
   savings, but I've found it helpful.

 - dbus defines subclasses of PyInts and PyLongs.  When porting to Python 3,
   all of these have to become subclasses of PyLongs, however for some of
   them, the exact hierarchy doesn't matter so much, so I've switched them to
   use PyLongObjects.

   Python 3.0 had a <intobject.h> compatibility header which I think would
   have been nice, but that's gone in Python 3.2.

   In Python 2, PyLongObject isn't defined unless you also #include
   <longintrepr.h>.  <Python.h> isn't enough.

 - The extension module interns a couple of strings.  In Python 2 this is
   PyString_InternFromString while in Python 3 it's
   PyUnicode_InternFromString.  I have the following macro for this:

   #ifdef PY3K
   #define INTERN PyUnicode_InternFromString
   #else
   #define INTERN PyString_InternFromString
   #endif

 - There are several places where PyArg_Parse*() wants to get a char*.
   Under Python 2, these just provide "s" codes and get passed a PyString.
   Under Python 3, I decided to allow either a bytes object or a utf-8 encoded
   unicode, but I always want to coerce it to a bytes internally, making it
   easy to extract the char*.

   I decided to switch the "s" codes to O& codes and add the following
   converter function:

   #ifdef PY3K
   #define RETURN_CLEANUP Py_CLEANUP_SUPPORTED
   #else
   #define RETURN_CLEANUP 1
   #endif

   int
   dbus_parse_bytes(PyObject *object, void *address)
   {
       PyObject *bytes;
       Py_ssize_t size;
       void *data;

       if (!object) {
           /* This is Python having a parse error, so free our reference. */
           Py_CLEAR(*(PyObject **)address);
           return 1;
       }
       if (PyBytes_Check(object)) {
           bytes = object;
           Py_INCREF(bytes);
       }
       else {
           if (!(bytes = PyUnicode_AsUTF8String(object)))
               return 0;
       }
       /* Embedded NULs are not allowed in dbus. */
       size = PyBytes_GET_SIZE(bytes);
       data = PyBytes_AS_STRING(bytes);
       if (size != (Py_ssize_t)strlen(data)) {
           PyErr_SetString(PyExc_TypeError, "embedded NUL character");
           Py_DECREF(bytes);
           return 0;
       }
       *(PyObject**)address = bytes;
       return RETURN_CLEANUP;
   }

   I think there's a potential for leaking these args under Python 2 when
   subsequent parse codes fail, because Py_CLEANUP_SUPPORTED isn't defined.
   I'm not sure there's anything that can be done about it, so hopefully it's
   rare enough not to matter in practice.

Things I haven't macro'd around:

 - PyCapsule vs PyCObject; I just #ifdef around the whole block of code.

 - A number of places want to check if something's a PyInt or a PyLong.  The
   PyInt checks can't be performed under Python 3, so I have some rather ugly
   #ifdefs sprinkled in various conditional.  (I suppose I could no-op
   PyInt_Check under Python 3).

 - Py_TPFLAGS_HAVE_WEAKREFS doesn't exist in Python 3 so I have to ifdef
   around setting the flags.  It might be nice if that was no-op'd in the
   compatibility header.

 - The changes to module inits are just a pain.  I'm not sure there's really
   anything you can do to make it nicer.  The C porting guides both on
   python.org and on python3porting.com provide some strategies, and I rolled
   my own slightly different approach based on those examples.

   I did define this:

   #ifdef PY3K
   #define RETURN_INITERROR return NULL
   #else
   #define RETURN_INITERROR return
   #endif

   just to make error condition returns a little easier to write.

 - Py_BuildValue() does not have a "y" code in Python 2, so you basically have
   to ifdef around that.

A few more things I ran across at the Python level:

 - For the Python code, I wanted to avoid 2to3, and was mainly successful with
   some liberal sprinkling of sys.version_info.major checks, and __future__
   imports (e.g. print_function, unicode_literals, and absolute_imports).
   Many of these might be nicer with your six module.

 - Long literals (i.e. trailing 'L's are a pain).

 - Metaclasses are a huge pain because the Python 3 syntax prevents
   compilation in Python 2, so you can't use sys.version_info.major checks
   alone.  Looks like six has a nice helper for this; I ended up using exec,
   but I think both cases would be rather painful if the derived class were
   anything more than a `pass` in the body.

 - iteritems() and friends are a pain.  In my case, I think they just weren't
   very useful, so I switched everything back to items() and such.

 - Similarly with xrange().

 - isSequenceType() is gone in Python 3.

 - Dealing with __next__() vs. next() methods.

Anyway, that's everything I kept notes on.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20111123/ce1167d5/attachment.pgp>

From dmalcolm at redhat.com  Thu Nov 24 16:22:34 2011
From: dmalcolm at redhat.com (David Malcolm)
Date: Thu, 24 Nov 2011 10:22:34 -0500
Subject: [Python-porting] [RELEASED] six 1.1
In-Reply-To: <20111123155915.2fd97971@limelight.wooz.org>
References: <CAPZV6o_zapLFbEw41mhqymo2t1dQ17Jd9Uj_9kp22ft+jm88sg@mail.gmail.com>
	<20111123101455.4d12402f@limelight.wooz.org>
	<CAPZV6o9d72v3gfL+qtngiRnx+KjNhTuPufEjcd+k4=+3wWNrdw@mail.gmail.com>
	<20111123155915.2fd97971@limelight.wooz.org>
Message-ID: <1322148155.2189.12.camel@surprise>

On Wed, 2011-11-23 at 15:59 -0500, Barry Warsaw wrote:
[...snip...]
> There are a bunch of little things I've found helpful while porting
> dbus-python.  [...snip...]

BTW, for porting dbus-python to py3k, did you see:
  https://bugs.freedesktop.org/show_bug.cgi?id=26420

Dave


From barry at python.org  Sun Nov 27 02:05:53 2011
From: barry at python.org (Barry Warsaw)
Date: Sat, 26 Nov 2011 20:05:53 -0500
Subject: [Python-porting] [RELEASED] six 1.1
In-Reply-To: <1322148155.2189.12.camel@surprise>
References: <CAPZV6o_zapLFbEw41mhqymo2t1dQ17Jd9Uj_9kp22ft+jm88sg@mail.gmail.com>
	<20111123101455.4d12402f@limelight.wooz.org>
	<CAPZV6o9d72v3gfL+qtngiRnx+KjNhTuPufEjcd+k4=+3wWNrdw@mail.gmail.com>
	<20111123155915.2fd97971@limelight.wooz.org>
	<1322148155.2189.12.camel@surprise>
Message-ID: <20111126200553.3b53dac6@limelight.wooz.org>

On Nov 24, 2011, at 10:22 AM, David Malcolm wrote:

>On Wed, 2011-11-23 at 15:59 -0500, Barry Warsaw wrote:
>[...snip...]
>> There are a bunch of little things I've found helpful while porting
>> dbus-python.  [...snip...]
>
>BTW, for porting dbus-python to py3k, did you see:
>https://bugs.freedesktop.org/show_bug.cgi?id=26420

Yep.  I did review the patch in the Fedora tracker, but I need to review this
again now that I'm farther along.  I still don't know whether Simon will like
my changes, but I will post my diff to that issue early next week.

I should note though that the test suite for the version in Ubuntu doesn't
fully pass even with Python 2.6 or 2.7, and it fails in my branch in the same
way.  I'd like to try to get that fixed, or at least see if the upstream
branch fails in the same way.  I am also seeing one test hang for reasons I
haven't figured out yet (but I have a suspicion ;).

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-porting/attachments/20111126/8c220eca/attachment.pgp>