[Python-Dev] Breaking up the stdlib (Was: release cadence)

[Steve Dower]

On 05Jul2016 1404, Brett Cannon wrote:
> Realizing that all of these are just proposals with no solid plan behind
> them, they are all predicated on moving to GitHub, and none of these are
> directly promoting releasing every module in the stdlib on PyPI as a
> stand-alone package with its own versioning, they are:
>
>  1.  Break the stdlib out from CPython and have it be a stand-alone repo
>  2. Break the stdlib up into a bunch of independent repos that when
>     viewed together make up the stdlib (Steve Dower did some
>     back-of-envelope grouping and pegged the # of repos at ~50)

Actually, I was meaning to directly promote releasing each module on 
PyPI as a standalone package :)

"Each module" here is at most the ~50 I counted (though likely many many 
fewer), which sounds intimidating until you realise that there are 
virtually no cross-dependencies between them and most would only depend 
on the core stdlib (which would *not* be a package on PyPI - you get 
most of Lib/*.py with the core install and it's fixed, while much of 
Lib/**/* is separately updateable).

Take email as an example. It's external dependencies (found by grep for 
"import") are abc, base64, calendar, datetime, functools, imghdr, os, 
quopri, sndhdr, socket, time, urllib.parse, uu, warnings. IMHO, only 
urllib has the slightest chance of being non-fixed here (remembering 
that "non-fixed" means upgradeable from PyPI, not that it's missing). 
The circular references (email<->urllib) would probably need to be 
resolved, but I think many would see that as a good cleanliness step anyway.

A quick glance suggests that both email and urllib are only using each 
other's public APIs, which means that any version of the other package 
is sufficient. An official release picks the two latest designated 
stable releases (this is where I'm imagining a requirements.txt-like 
file in the core repository) and bundles them both, and then users can 
update either package on its own. If email makes a change that requires 
a particular change to urllib, it adds a version constraint, which will 
force both users and the core requirements.txt file to update both 
together (this is probably why the circular references would need 
breaking...)

Done with care and even incrementally (e.g. just the provisional modules 
at first), I don't think this is that scary a proposition. It's not 
strictly predicated on moving to github or to many repositories, but if 
we did decide to make a drastic change to the repository layout (which I 
think this requires at a minimum, at least for our own sanity), doing it 
while migrating VCS at least keeps all the disruption together.

Cheers,
Steve