[Linux-SIG] PEP 476 and backward incompatibilities for distros

[Barry Warsaw]

And now for something completely different...

In Ubuntu, as probably with other distros, we have long-term support (LTS)
releases.  We have a process called stable release upgrades (SRU) to allow us
to systematically update packages in existing, supported releases and we[*]
try to keep LTS versions of Python current with upstream.  The policy of
breaking backward compatibility, even for security issues, can cause us all
kinds of pain.  I know these types of changes are controversial even upstream,
but they are very problematic for distros.  There's even talk about
disallowing point updates for Python in LTS releases because of this.  That
does our users a disservice because they do want all the other great bug fixes
in point releases.

A current example is captured in this bug:

https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1500768

which is related to PEP 476

https://www.python.org/dev/peps/pep-0476/

Sadly, I did not follow the discussion on PEP 476 at the time, and I won't
argue in this mailing list whether it was a good idea or not.  Here I want to
talk about making such changes friendlier to downstream consumers such as
Ubuntu.

The change showed up in Ubuntu 14.04 LTS when Python 3.4.3 was uploaded.  This
backward incompatible change broke several other packages and user complaints
started coming in.  We eventually reverted the upload, but not until after
lots of people probably upgraded to get it.  So now their systems are broken.

I know that urllib3/requests upstream was updated to work with this change,
but that is problematic for SRUs because we don't want to just backport the
latest versions --with new APIs-- of those into the trusty-updates channel.
Those new versions may themselves cause cascading incompatibilities.  We could
try to cherry pick the fixes in those packages, but that's a lot of additional
work, and it doesn't help any other breakages caused by PEP 476.

Python need to do a better job of not breaking backward compatibility, or
making it easy to identify and omit such compatibility breaking changes when
pulled in by downstreams.  For example, Misc/NEWS says only this in the What's
new in 3.4.3. section:

- Issue #22417: Verify certificates by default in httplib (PEP 476).

That's way too easy to miss, doesn't include any of the backward compatibility
break implications, and doesn't really provide much help in narrowing down the
commit or commits related to this.  The workarounds described in the PEP's
Opting Out section isn't really useful either.  If we applied this to the
Ubuntu version, how would people know what to do to opt back in?

These kinds of incompatible changes in Python stable releases give Python a
bad reputation, and make folks very wary about just adopting such any
maintenance release.  That's not a good situation to be in when your users
*also* want the latest updates to get other, non breaking fixes.

It's a trade-off for sure.  It's not necessarily a bad thing to want to
improve your user's default security.  It *might* be a bad thing to do that in
stable releases.  It's *definitely* a bad thing to do it in a way that breaks
backward compatibility, or is not very clearly marked and easily ignored by
distros which need to make different decisions.

Cheers,
-Barry

[*] well Matthias does a considerable amount of the actual work.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/linux-sig/attachments/20151008/5a2badd7/attachment.sig>