The real problem with Python 3 - no business case for conversion (was "I strongly dislike Python 3")

[David Cournapeau]

On Wed, Jul 7, 2010 at 10:55 PM, Carl Banks <pavlovevidence at gmail.com> wrote:
> On Jul 7, 1:31 am, Paul McGuire <pt... at austin.rr.com> wrote:
>> On Jul 6, 3:30 am, David Cournapeau <courn... at gmail.com> wrote:> On Tue, Jul 6, 2010 at 4:30 AM, D'Arcy J.M. Cain <da... at druid.net> wrote:
>>
>> > One thing that would be very useful is how to maintain something that
>> > works on 2.x and 3.x, but not limiting yourself to 2.6. Giving up
>> > versions below 2.6 is out of the question for most projects with a
>> > significant userbase IMHO. As such, the idea of running the python 3
>> > warnings is not so useful IMHO - unless it could be made to work
>> > better for python 2.x < 2.6, but I am not sure the idea even makes
>> > sense.
>>
>> This is exactly how I felt about my support for pyparsing, that I was
>> trying continue to provide support for 2.3 users, up through 3.x
>> users, with a single code base.  (This would actually have been
>> possible if I had been willing to introduce a performance penalty for
>> Python 2 users, but performance is such a critical issue for parsing I
>> couldn't justify it to myself.)  This meant that I had to constrain my
>> implementation, while trying to incorporate forward-looking support
>> features (such as __bool__ and __dir__), which have no effect on older
>> Python versions, but support additions in newer Pythons.  I just
>> couldn't get through on the python-dev list that I couldn't just
>> upgrade my code to 2.6 and then use 2to3 to keep in step across the
>> 2-3 chasm, as this would leave behind my faithful pre-2.6 users.
>>
>> Here are some of the methods I used:
>>
>> - No use of sets.  Instead I defined a very simple set simulation
>> using dict keys, which could be interchanged with set for later
>> versions.
>>
>> - No generator expressions, only list comprehensions.
>>
>> - No use of decorators.  BUT, pyparsing includes a decorator method,
>> traceParseAction, which can be used by users with later Pythons as
>> @traceParseAction in their own code.
>>
>> - No print statements.  As pyparsing is intended to be an internal
>> module, it does no I/O as part of its function - it only processes a
>> given string, and returns a data structure.
>>
>> - Python 2-3 compatible exception syntax.  This may have been my
>> trickiest step.  The change of syntax for except from
>>
>>     except ExceptionType, ex:
>>
>> to:
>>
>>     except ExceptionType as ex:
>>
>> is completely forward and backward incompatible.  The workaround is to
>> rewrite as:
>>
>>     except ExceptionType:
>>         ex = sys.exc_info()[0]
>>
>> which works just fine in 2.x and 3.x.  However, there is a slight
>> performance penalty in doing this, and pyparsing uses exceptions as
>> part of its grammar success/failure signalling and backtracking; I've
>> used this technique everywhere I can get away with it, but there is
>> one critical spot where I can't use it, so I have to keep 2 code bases
>> with slight differences between them.
>>
>> - Implement __bool__, followed by __nonzero__ = __bool__.  This will
>> give you boolean support for your classes in 2.3-3.1.
>>
>> - Implement __dir__, which is unused by old Pythons, but supports
>> customization of dir() output for your own classes.
>>
>> - Implement __len__, __contains__, __iter__ and __reversed__ for
>> container classes.
>>
>> - No ternary expressions.  Not too difficult really, there are several
>> well-known workarounds for this, either by careful use of and's and
>> or's, or using the bool-as-int to return the value from
>> (falseValue,trueValue)[condition].
>>
>> - Define a version-sensitive portion of your module, to define
>> synonyms for constants that changed name between versions.  Something
>> like:
>>
>>     _PY3K = sys.version_info[0] > 2
>>     if _PY3K:
>>         _MAX_INT = sys.maxsize
>>         basestring = str
>>         _str2dict = set
>>         alphas = string.ascii_lowercase + string.ascii_uppercase
>>     else:
>>         _MAX_INT = sys.maxint
>>         range = xrange
>>         _str2dict = lambda strg : dict( [(c,0) for c in strg] )
>>         alphas = string.lowercase + string.uppercase
>>
>> The main body of my code uses range throughout (for example), and with
>> this definition I get the iterator behavior of xrange regardless of
>> Python version.
>>
>> In the end I still have 2 source files, one for Py2 and one for Py3,
>> but there is only a small and manageable number of differences between
>> them, and I expect at some point I will move forward to supporting Py3
>> as my primary target version.  But personally I think this overall
>> Python 2-3 migration process is moving along at a decent rate, and I
>> should be able to make my switchover in another 12-18 months.  But in
>> the meantime, I am still able to support all versions of Python NOW,
>> and I plan to continue doing so (albeit "support" for 2.x versions
>> will eventually mean "continue to offer a frozen feature set, with
>> minimal bug-fixing if any").
>>
>> I realize that pyparsing is a simple-minded module in comparison to
>> others: it is pure Python, so it has no issues with C extensions; it
>> does no I/O, so print-as-statement vs. print-as-function is not an
>> issue; and it imports few other modules, so the ones it does have not
>> been dropped in Py3; and overall it is only a few thousand lines of
>> code.  But I just offer this post as a concrete data point in this
>> discussion.
>
> Thanks for the helpful post.  However it looks like, other than the
> except syntax, all of these are things you're already doing to support
> 2.3 to 2.6, so it seems suggest supporting 3.1 and 2.6 is maybe a
> little more work than supporting 2.3 and 2.6.
>
> We all know that Python is quite successful even with a history of
> backward-incompatible changes and feature additions and even minor
> paradigm shifts.  What I'm interested in is, what things about the 2
> to 3 transition is harder than already exists?

The 2->3 transition is harder because backward incompatibilities are
much more pervasive. In Numpy, adapting the code for python 2.6 was
mostly trivial, for example (removing variables named "as" or "with").
For 3.x, it required much more work.

The question really is one of cost vs benefit: for any new version of
2.x, supporting it was very useful because new python 2.x would become
the default of popular linux distribution, and that's what most casual
windows users would get since it was the default download.

Python 3.x required much more work, for a much smaller perceived
benefit as far as I am concerned (I don't claim to represent anyone
but myself here for numpy transition to 3.x). In particular, assuming
the main argument of python 3.x is a cleaner language, it evaporates
for projects which need to support both 2.x and 3.x projects. Unless
you maintain 2 codebases, this means the project code is less clean,
not cleaner because of compatibility requirements.

As most people I guess, I find discussion about print vs print()
completely pointless - certainly, this is the simplest and most
trivial change to apply to a codebase. But I think python 3.x needs to
bring more tangible advantages to make the cost more bearable. I am
hopeful that python 3.4 or 3.5 will have features/improvements which
will make this worthwhile, but 3.1, not so much,

David