[Python-Dev] Policy Decisions, Judgment Calls, and Backwards Compatibility (was Re: splitext('.cshrc'))

[glyph at divmod.com]

martin at v.loewis.de wrote:
>That assumes there is a need for the old functionality. I really don't
>see it (pje claimed he needed it once, but I remain unconvinced, not
>having seen an actual fragment where the old behavior is helpful).

This passage is symptomatic of the thing that really bothers me here.  I 
have rarely used splitext before (since what I really want is 
determineMIMEType(), and that is easier to implement with one's own 
string-munging) and certainly won't use it again after following this 
discussion.

The real issue I have here is one of process.  Why is it that PJE (or 
any python user who wishes their code to keep working against new 
versions of Python) must frequent python-dev and convince you (or 
whatever Python developer might be committing a patch) of each use-case 
for old functionality?  I would think that the goal here would be to 
keep all the old Python code running which it is reasonably possible to, 
regardless of whether the motivating use-cases are understood or not. 
It is the nature of infrastructure code to be used in bizarre and 
surprising ways.

My understanding of the current backwards-compatibility policy for 
Python, the one that Twisted has been trying to emulate strictly, is 
that, for each potentially incompatible change, there will be:

* at least one release with a pending deprecation warning and new, 
better API
* at least one release with a deprecation warning
* some number of releases later, the deprecated functionality is removed

I was under the impression that this was documented in a PEP somewhere, 
but scanning the ones about backwards compatibility doesn't yield 
anything.  I can't even figure out why I had this impression.  *Is* 
there actually such a policy?

If there isn't, there really should be.  Deciding each one of these 
things on a case-by-case basis leaves a lot of wiggle room for a lot of 
old code to break in each release.  For example, if you put me in charge 
of Python networking libraries and I simply used my judgment about what 
usages make sense and which don't, you might find that all the 
synchronous socket APIs were mysteriously gone within a few releases... 
;-)

Perhaps policy isn't the right way to solve the problem, but neither is 
asking every python application developer to meticulously follow and 
participate in every discussion on python-dev which *might* affect their 
code.

As recently as last week, Guido commented that people build mental 
models of performance which have no relation to reality and we must rely 
on empirical data to see how things *really* perform.  This is a common 
theme both here and anywhere good software development practices are 
discussed.

These broken mental models are not limited to performance.  In 
particular, people who develop software have inaccurate ideas about how 
it's really used.  I am avoiding being specific to Python here because 
I've had a similarly broken idea of how people use Twisted, heard 
extremely strange ideas from kernel developers about the way programs in 
userland behave, and don't get me started on how C compiler writers 
think people write C code.

If Python is not going to have an extremely conservative (and 
comprehensive, and strictly enforced) backwards-compatibility policy, we 
can't rely on those mental models as a way of determining what changes 
to allow.  One way to deal with the question of "how do people really 
use python in the wild" is to popularize the community buildbots and 
make it easy to submit projects so that at least we have a picture of 
what Python developers are really doing.

Buildbot has a "build this branch" feature which could be used to settle 
these discussions more rapidly, except for the fact that the community 
builders are currently in pretty sad shape:

    http://www.python.org/dev/buildbot/community/all/

By my count, currently only 9 out of 22 builders are passing.  The 
severity of these failures varies (many of the builders are simply 
offline, not failing) but they should all be passing.  If they were, 
rather than debating use-cases, we could at *least* have someone check 
this patch into a branch, and then build that branch across all these 
projects to see if any of them failed.

Unfortunately open source code is quite often better tested and 
understood than the wider body of in-house and ad-hoc Python code out 
there, so it will be an easier target for changes like this than 
reality.  I don't really have a problem with that though, because it 
creates a strong incentive for Python projects to both (A) be open 
source and (B) write comprehensive test suites, both of which are useful 
goals for many other reasons.

In the past I've begged off of actually writing PEPs because I don't 
have the time, but if there is interest in codifying this I think I 
don't have the time *not* to write it.  I'd prefer to document the 
pending/deprecate/remove policy first, but I can write up something more 
complicated about community buildbots and resolving disputes around 
potential breakages if there is actually no consensus about that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070309/9e4d0d57/attachment.htm