[python-committers] "Gratuitous"? incompatibilities in the "fix only" releases

[R. David Murray]

On Wed, 29 Jul 2015 13:41:09 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/29/2015 1:01 PM, Robert Collins wrote:
> > On 30 July 2015 at 04:50, Guido van Rossum <guido at python.org> wrote:
> >> I believe that in this particular case, the bug was fixed (by tightening the
> >> requirements for headers) because the bug can lead to security
> >> vulnerabilities. I think you can find more by Googling for keywords like
> >> "http header injection". The more recent Python 2.7 bugfix releases have
> >> specific exemptions from the backwards compatibility requirements for
> >> security fixes -- because their lifespan will still be many years (EOL of
> >> 2.7 is summer 2020).
> >
> > Yeah - this is a security issue, and unfortunately its one that can
> > break programs [or rather, expose how they were broken already at an
> > earlier and less susceptible point].
> >
> > As a new committer, I'd like to double check my understanding of the policy:
> >
> > https://docs.python.org/devguide/devcycle.html#maintenance-branches
> > "...
> > The only changes allowed to occur in a maintenance branch without
> > debate are bug fixes. Also, a general rule for maintenance branches is
> > that compatibility must not be broken at any point between sibling
> > minor releases (3.4.1, 3.4.2, etc.).
> 
> Since bug fixes break code that depends on the bug (as happened in this 
> case), the second rule appears to be written too strongly.  It really 
> needs a short paragraph.  Bug fixes should only break code depending on 
> the bug. Bug fixes must not change existing non-buggy features and 
> should not introduce new features.  Non-security bug fixes that break 
> too much code deemed to be reasonable are sometimes deferred to the next 
> release.

No, I don't think it is too strong.  Normally even code that depends on
the bug should not be broken.

The calculus is necessarily a work of art:  how likely is this bug fix
to break working code?  If the bug fixes something that previously
raised an error, it is (usually) a no-brainer.  If it fixes an erroneous
result it is a judgement call based on how likely fixing it is to break
working code, but usually it would get fixed.  If it fixes an erroneous
behavior it is again a judgement call, but weighted toward not going in
to a maintenance release.  Bugs judged too likely to break working code
get fixed in feature releases (and mentioned in What's New).

>  > For both rules, only rare
> > exceptions are accepted and must be discussed first."
> >
> > Where should these things be discussed? I've been discussing with
> > other committers on the issues in the issue tracker. Is this
> > sufficient? What is the social norm?

Usually the bug tracker is enough, with escalation to python-dev if
there is a core-committer disagreement.  For security issues, the
security team should get involved in any decision, and their decision is
pretty much final.  I believe they automatically get notified if
'security' is selected for behavior...if not we should make it so.

> Feature additions like adding a new parameter to fix a bug should be 
> discussed on pydev.  For instance, difflib.SequenceMatcher gained the 
> autojunk parameter in 2.7.1.  I believe the pydev discussion included 
> "Is the issue a bug?" (yes) and "Does it need fixing in the current 
> release?" (yes, it generated multiple bug reports). I believe being 
> early in the long 2.7.x series and the last change to fix in 2.x played 
> a role.

This would be a very exceptional case.  I remember a discussion, I don't
remember the approval of an API change.  Changing the API in a minor
release *except* for security issues is not normally acceptable.
I'd be curious to read the reasoning behind this one.

So, the rule in the devguide is accurate, except that it should note
that exceptions are made for fixing security related issues.  That is,
the calculus about not breaking working code is given a lot less weight,
because the goal of the bug fix is to patch a security hole.  We find
it OK to break working code in order to make that code less vulnerable
to a known attack.  Although still try very hard *not* to break working
code, if at all possible.

--David