[Catalog-sig] Deprecate External Links

[Nick Coghlan]

On Thu, Feb 28, 2013 at 5:01 PM, Donald Stufft <donald.stufft at gmail.com> wrote:
> I'm glad the next set of Metadata won't have external links, however
> even if it showed up tomorrow it's going to be a long time until
> people are completely migrated to it. Furthermore you estimate
> months but the first phase will have positive benefits right away, namely
> that it will prompt people to start uploading their packages better
> increasing
> the security and reliability of the current system. And finally while I'm
> glad to see forward movement It's been said before not to bother
> making a fix to the existing system because X was going to happen
> soon, in the past i was distutils2/packaging, now it's PEP426/packaging.
> While I have every hope and I believe it will happen this time, the
> past has made me worry about holding off on good incremental
> improvements to the current infra.

Pissing off the maintainers off packages that currently rely on
external hosting by telling them they have to change their release
processes if they want to keep releasing software on PyPI and have
their users actually be able to download it is *not* a good idea,
especially when we're about to ask them to upgrade their build chains
for other reasons (including both security and reliability).

Working on the installation tools and getting them to complain about
external links is a *fine* idea. It doesn't break anything, but
maintainers will start fielding questions from their users asking
"Hey, why am I getting this warning when I install your project?".
Working on the upload tools and having them *warn* distributors that
self-hosting is problematic is also a good idea, as is exploring PJE's
suggestions about refining the set of URLs that PyPI currently
publishes

However, getting PyPI to effectively *break uploads* of projects that
rely on external links at this point in time is *not* a good idea: we
should NOT mess with people's existing build and upload processes
lightly, as any such changes burn up a *lot* of community good will,
and that's not something in great supply when it comes to Python's
software distribution infrastructure. All current generation
infrastructure should continue to work without modification on both
the upload side *and* the download side (although, as noted above,
it's highly desirable for both the upload side and the download side
to be updated to warn users about any reliance on insecure legacy
behaviour).

I expect a similar rollout in the transition to the next generation
metadata format and distribution infrastructure - initially download
tools will support both formats, emitting a warning when falling back
to the legacy distribution infrastructure, then they will start
requiring an option to enable fallback to legacy mode, and eventually
there will be released installation tools that don't support the
legacy distribution infrastructure at all (such as any default
installer included in the standard library).

For *next* generation infrastructure, it's our job as system designers
to sell it to potential users (primarily "everyone writing software
which they publish on PyPI, or at least the authors of the toolchains
used for that publication", but also to consumers of that software).
The distutils2 team failed, in large part because their proposal
required radical changes to the way people published Python software.
I have deliberately moderated that in the way I have approached PEP
426 - if people can't generate the new metadata with only minor
changes to their current processes, it isn't going to fly, and any
trade-offs (such as the loss of external hosting support), need to be
bought with corresponding benefits (such as guaranteed correct
pre-release handling, solid Python version declarations, a clean
post-install hook design, and, hopefully, a vastly improved rich
metadata publication system for PyPI, probably based on TUF).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia