[Linux-SIG] PEP 476 and backward incompatibilities for distros

[Nick Coghlan]

On 9 Oct 2015 02:29, "Barry Warsaw" <barry at python.org> wrote:
>
> And now for something completely different...
>
> In Ubuntu, as probably with other distros, we have long-term support (LTS)
> releases.  We have a process called stable release upgrades (SRU) to
allow us
> to systematically update packages in existing, supported releases and
we[*]
> try to keep LTS versions of Python current with upstream.  The policy of
> breaking backward compatibility, even for security issues, can cause us
all
> kinds of pain.  I know these types of changes are controversial even
upstream,
> but they are very problematic for distros.  There's even talk about
> disallowing point updates for Python in LTS releases because of this.

This is the RHEL/CentOS backports-only model for long term support, and I
think it's the only way to offer good commercial stability guarantees
without placing an undue maintenance burden on community volunteers.
Rebasing in long term support releases should be the exception rather than
the rule.

> That
> does our users a disservice because they do want all the other great bug
fixes
> in point releases.

This is a classic case of a conflict of interest between commercial
redistributors and the upstream community.

For upstream, working on PEP 466 persuaded me that the priority needs to be
on providing safe defaults in order to minimise risks for new users, and
for insufficiently paranoid existing users.

The burden then falls on commercial redistributors to effectively manage
backwards incompatible changes on behalf of our customers, rather than
expecting upstream to prioritise our interests over those of the rest of
the community.

> A current example is captured in this bug:
>
> https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1500768
>
> which is related to PEP 476
>
> https://www.python.org/dev/peps/pep-0476/
>
> Sadly, I did not follow the discussion on PEP 476 at the time, and I won't
> argue in this mailing list whether it was a good idea or not.  Here I
want to
> talk about making such changes friendlier to downstream consumers such as
> Ubuntu.
>
> The change showed up in Ubuntu 14.04 LTS when Python 3.4.3 was uploaded.
This
> backward incompatible change broke several other packages and user
complaints
> started coming in.  We eventually reverted the upload, but not until after
> lots of people probably upgraded to get it.  So now their systems are
broken.

The model we've adopted to deal with this on the Fedora/RHEL/CentOS side of
things is:

* Fedora will break things sometimes. If that's not acceptable to a given
user, then Fedora probably isn't the distro for them
* RHEL and (by extension) CentOS, are far more selective about the updates
they bring in. This is more expensive to maintain on the redistributor side
(since a lot of changes need to be backported to earlier upstream base
releases), but means there's an additional layer of review and change
management around potentially breaking changes, so we don't need to argue
against improvements to default behaviour upstream.
* Software Collections then make faster moving rebases readily available
atop the stable RHEL/CentOS layer. They're more stable than Fedora, but
also have major releases (with potential compatibility breaks) more often
than RHEL.

> I know that urllib3/requests upstream was updated to work with this
change,
> but that is problematic for SRUs because we don't want to just backport
the
> latest versions --with new APIs-- of those into the trusty-updates
channel.
> Those new versions may themselves cause cascading incompatibilities.  We
could
> try to cherry pick the fixes in those packages, but that's a lot of
additional
> work, and it doesn't help any other breakages caused by PEP 476.

Providing a better migration path for cases where commercial support
obligations apply is what the proposed PEP 493 recommendations are about:
https://www.python.org/dev/peps/pep-0493/

> Python need to do a better job of not breaking backward compatibility, or
> making it easy to identify and omit such compatibility breaking changes
when
> pulled in by downstreams.

s/Python/vendors commercially redistributing Python/

> For example, Misc/NEWS says only this in the What's
> new in 3.4.3. section:
>
> - Issue #22417: Verify certificates by default in httplib (PEP 476).
>
> That's way too easy to miss, doesn't include any of the backward
compatibility
> break implications, and doesn't really provide much help in narrowing
down the
> commit or commits related to this.  The workarounds described in the PEP's
> Opting Out section isn't really useful either.  If we applied this to the
> Ubuntu version, how would people know what to do to opt back in?

This is on redistributors to handle - if we want more or different
documentation in cases like these, then we need to be paying people to
spend more time on representing our interests (and those of our users) in
CPython upstream development, rather than focusing our efforts solely on
integration work for our respective operating systems.

> These kinds of incompatible changes in Python stable releases give Python
a
> bad reputation, and make folks very wary about just adopting such any
> maintenance release.  That's not a good situation to be in when your users
> *also* want the latest updates to get other, non breaking fixes.

The perpetuation of bad default behaviours in the name of backwards
compatibility is a large part of what turned our industry into the insecure
user hostile mess that it is today, though.

> It's a trade-off for sure.  It's not necessarily a bad thing to want to
> improve your user's default security.  It *might* be a bad thing to do
that in
> stable releases.  It's *definitely* a bad thing to do it in a way that
breaks
> backward compatibility, or is not very clearly marked and easily ignored
by
> distros which need to make different decisions.

My own focus for these changes has been on Python 2.7, and the changes are
far more clearly marked there:
https://docs.python.org/2/whatsnew/2.7.html#py27-maintenance-enhancements

I never looked at how the transition was handled for 3.4 myself (as I was
mainly concerned about the impact on RHEL & CentOS, rather than the Python
3 stacks in Fedora & SCLs)

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/linux-sig/attachments/20151009/a510dc25/attachment.html>