[Python-Dev] PEP 407 / splitting the stdlib

[Nick Coghlan]

On Wed, Jan 18, 2012 at 10:30 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Splitting the stdlib:
> - requires someone to do the splitting (highly non-trivial given the
> interactions of some modules with interpreter details or low-level C
> code)
> - requires setting up separate resources (continuous integration with N
> stdlib versions and M interpreter versions, for example)
> - requires separate maintenance and releases for the stdlib (but with
> non-trivial interaction with interpreter maintenance, since they will
> affect each other and must be synchronized for Python to be usable at
> all)
> - requires more attention by users since there are now *two* release
> schedules and independent version numbers to track

Did you read what I actually proposed? I specifically *didn't* propose
separate stdlib releases (for all the reasons you point out), only
separate date based stdlib *versioning*. Distribution of the CPython
interpreter + stdlib would remain monolithic, as it is today. Any
given stdlib release would only be supported for the most recent
language release. The only difference is that between language
releases, where we currently only release maintenance builds, we'd
*also* release a second version of each maintenance build with an
updated standard library, along with an alpha release of the next
language version (with the last part being entirely optional, but I
figured I may as well make the suggestion since I like the idea to
encourage getting syntax updates and the like out for earlier
experimentation).

When you initially pitched the proposal via email, you didn't include
the "language moratarium applies to interim releases" idea. That one
additional suggestion makes the whole concept *much* more appealing to
me, but I only like it on the condition that we decouple the stdlib
versioning from the language definition versioning (even though I
recommend we only officially support very specific combinations of the
two). My suggestion is really just a concrete proposal for
implementing Ezio's idea of only bumping the Python version for
releases with long term support, and using some other mechanism to
distinguish the interim releases.

So, assuming a 2 year LTS cycle, the released versions up to February
2015 with my suggestion would end up being:

>From the default branch:
Python 3.3.0 + stdlib 12.08.0  (~August 2012)
Python 3.4.0a1 + stdlib 14.08.0a1  (~February 2013)
Python 3.4.0a2 + stdlib 14.08.0a2 (~August 2013)
Python 3.4.0a3 + stdlib 14.08.0a3  (~February 2014)
Python 3.4.0a4 + stdlib 14.08.0a4  (~2014)
Python 3.4.0b1 + stdlib 14.08.0b1  (~2014)
Python 3.4.0b2 + stdlib 14.08.0b2  (~2014)
Python 3.4.0c1 + stdlib 14.08.0c1  (~2014)
Python 3.4.0 + stdlib 14.08  (~August 2014)
Python 3.5.0a1 + stdlib 16.08.0a1  (~February 2015)

>From the 3.3 maintenance branch (these are maintenance updates to the
"LTS" release):
Python 3.3.1 + stdlib 12.08.1  (~February 2013)
Python 3.3.2 + stdlib 12.08.2  (~August 2013)
Python 3.3.3 + stdlib 12.08.3  (~February 2014)
Python 3.3.4 + stdlib 12.08.4  (~August 2014) (and 3.3 branch enters
security patch only mode)

>From the 3.4 maintenance branch (these are maintenance updates to the
"LTS" release):
Python 3.4.1 + stdlib 14.08.1  (~February 2015)

>From the stdlib feature development branch (these are the new interim
releases with standard library updates only as proposed by PEP 407):
Python 3.3.1 + stdlib 13.02.0  (~February 2013)
Python 3.3.2 + stdlib 13.08.0  (~August 2013)
Python 3.3.3 + stdlib 14.02.0  (~February 2014) (only upgrade path
from here is to make the jump to 3.4.0)
-- 3.4.0 + 12.08.0 is released from default branch --
Python 3.4.1 + stdlib 15.02.0  (~February 2015)

If we have to make "brown paper bag" releases for the maintenance or
stdlib branches then the micro versions get bumped - the date based
version of the standard library versions relates to when that
particular *API* was realised, not when bugs were last fixed in it. If
a target release date slips, then the stdlib version would be
increased accordingly (cf. Ubuntu 6.06).

Yes, we'd have an extra set of active buildbots to handle the stdlib
branch, but a) that's no harder than creating the buildbots for a new
maintenance branch and b) the interim release proposal will need to
separate language level changes from stdlib level changes *anyway*.

As far as how sys.version checks would be updated, I would propose a
simple API addition to track the new date-based standard lib
versioning: sys.stdlib_version. People could choose to just depend on
a specific Python version (implicitly depending on the stdlib version
that was originally shipped with that version of CPython), or they may
instead decide to depend on a specific stdlib version (implicitly
depending on the first Python version that was shipped with that
stdlib).

The reason I like this scheme is that it allows us (and users) to
precisely track the things that can vary at the two different rates.
At least the following would still be governed by changes in the first
two fields of sys.version (i.e. the major Python version):
    - deprecation policy
    - language syntax
    - compiler AST
    - C ABI stability
    - Windows compilation suite and C runtime version
    - anything else we decide to link with the Python language version
(e.g. default pickle protocol)

However, the addition of date based stdlib versioning would allow us
to clearly identify the new interim releases proposed by PEP 407
*without* mucking up all those things that are currently linked to
sys.version and really *shouldn't* be getting updated every 6 months.
Users get a clear guarantee that if they follow the stdlib updates
instead of the regular maintenance releases, they'll get nice new
features along with their bug fixes, but no new deprecations or
backwards incompatible API changes. However, they're also going to be
obliged to transition to each new language release as it comes out if
they want to continue getting security updates.

Basically, what it boils down to is that I'm now +1 on the general
proposal in the PEP, *so long as*:
1. We get a separate Hg branch for "stdlib only" changes and default
becomes the destination specifically for "language update" changes
(with the latter being a superset of the former)
2. The proposed "interim releases" are denoted by a new date-based
sys.stdlib_version field and sys.version retains its current meaning
(and slow rate of change)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia