[Distutils] Need for respect (was: PEP 438, pip and --allow-external)

[Nick Coghlan]

On 12 May 2014 21:34, M.-A. Lemburg <mal at egenix.com> wrote:
> Think about it: PyPI has become a great hosting platform in the last
> year, it's attractive to host packages on the platform and this also
> shows in the number of package authors that have decided to switch
> over to PyPI for hosting.
>
> The ones that don't, will have good reasons not to host on PyPI.
> We need to respect those reasons, not question them, and,
> if possible, improve the infrastructure to make it more inviting
> for them to join in.
>
> We should also reach out to the package authors that currently
> do not host on PyPI to find out what their reasons are and
> whether we can do something to help convince them.
>
> Note: I haven't yet read the whole thread on the distutils list,
> but do want everyone to keep the above in mind when discussing
> the topic.

When you get to the end of that thread, you'll find that we reached
two main conclusions:

1. Wanting to avoid US export laws is still an excellent reason for
not wanting to host files directly on PyPI (and we really only need
one such reason to accept external hosting as a use case that needs to
continue to be handled)*

2. The user experience of the current external hosting arrangements
really is incredibly poor, especially since there are *two* competing
external hosting mechanisms (the verifiable link spidering and
specifying additional index pages) and the implementation requirements
for the link spidering are also limiting pip's ability to provide
decent error messages for some failure modes.

To reduce user confusion (and simplify the pip implementation so it
can start providing better error messages), we'd like to consolidate
that down to just one external hosting mechanism over the next couple
of pip releases: additional custom index pages, as that's the more
powerful and flexible of the two systems, as well as the one that's
easiest to explain. That will mean a few different ideas need to be
explored/defined:

1. A deprecation process and timeline for the link spidering mechanism
(including a grandfather clause for existing packages)
2. An improved user experience for discovery of custom index server
requirements via PyPI
3. A simple process for publishing custom index pages (perhaps these
could even be PyPI hosted?)
4. A simple client configuration process for custom index servers
(perhaps with the ability to get custom servers pre-approved by the
pip developers and included in the default pip configuration)
5. Exploring the possibility of the PSF (or, if necessary, another
trusted entity incorporated outside the US) providing a secondary
hosting service located in Europe which would allow non-US users to
publish packages without needing to agree to be subject to US export
restrictions

Donald is planning to start working on a PEP this week to propose this
transition from implicit link spidering to an explicitly federated
model that is closer to the way Linux distros have historically worked
(one where you can register multiple trusted package sources locally,
and the installer is preconfigured with a set of trusted sources, but,
aside from mirroring networks, there's no way for one source to
implicitly delegate trust to another location).

Regards,
Nick.

P.S. *Regarding the reasons for external hosting, I also acknowledge
that there are currently some understandable concerns with the current
wording of the distribution license terms for PyPI, but I believe that
particular issue is best addressed by clarifying the wording of the
terms to make it clear they don't override the packaging licensing
terms in general, but are instead a supplemental license ensuring that
PyPI mirroring, along with automated installation and use of packages
is always permitted for all uploaded packages.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia