[Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

[M.-A. Lemburg]

On 27.08.2015 03:24, Donald Stufft wrote:
> While developing Warehouse, one of the things I wanted to get done was a final ruling on PEP 470. With that in mind I’d like to bring it back up for discussion and hopefully ultimately a ruling.
> 
> Their are two major differences in this version of PEP 470, and I’d like to point them out explicitly.
> 
> Removal of the “External Repository Discover” feature. I’ve been thinking about this for awhile, and I finally removed it. I’ve always been uncomfortable with this feature and I finally realized why it was. Essentially, the major use case for not hosting things on PyPI that I think PyPI can reasonably be expected to accommodate is people who cannot publish their software to the US for various reasons. At the time I came up with the solution I did, It was an attempt to placate the folks who were against PEP 470 while assuming very few people would ever actually use it, essentially a junk feature to push the PEP through. I think that the feature itself is a bad feature and I think it presents a poor experience for people who want to use it, so I’ve removed it from the PEP and instead focused the PEP on explicitly recommending that all installers should implement the ability to specify multiple repositories and deprecating and removing the ability for finding anything but file
 s hoste
d by the repository itself on /simple/.

This feature was part of a compromise to reach consensus on the removal
of external hosting. While I don't think the details of the repository discovery
need to be part of PEP 470, I do believe that the PEP should continue
to support the idea of having a way for package managers to easily
find external indexes for a particular package and not outright reject it.

Instead, the PEP should highlight this compromise and defer it to a
separate PEP.

More comments:

* The user experience argument you give in the PEP 470 for rejecting the
idea is not really sound: the purpose of the discovery feature is to
provide a *better user experience* than an error message saying that a package
cannot be found and requiring the user to do research on the web to find
the right URLs. Package managers can use the information about the
other indexes they receive from PyPI to either present them to the user
to use/install them or to even directly go there to find the packages.

* The section on data sovereignty should really be removed or reworded.
PEPs should be neutral and not imply political views, in particular not
make it look like the needs of US users of PyPI are more important then
those of non-US users. Using "poor user experience" as argument here is
really not appropriate.

  PyPI is a central part of the Python community infrastructure and
should be regarded as a resource for the world-wide community.
There is no reason to assume that we cannot have several PyPI
installations around the world to address any such issues.

 * It is rather unusual to have a PEP switch from a compromise solution
to a rejection of the compromise this late in the process.

I will soon be starting a PSF working group to address some of
the reasons why people cannot upload packages to PyPI. The intent
is to work on the PyPI terms to make them more package author
friendly. Anyone interested to join ?

> I recognize this is a regression for anyone who *does* have concerns with uploading their projects to a server hosted in the US. If there is someone that has this concern, and is also willing to put in the effort and legwork required, I will happily collaborate with them to design a solution that both follows whatever legal requirements they might have, as well as provides a good experience for people using PyPI and pip. I have some rough ideas on what this could look like, but I think it’s really a separate discussion since I believe externally hosted files like we were is an overall bad experience for people and is largely a historic accident from how PyPI and Python packaging has evolved. I don’t want to derail this thread or PEP exploring these ideas (some of which I don’t even know if they would satisfy the requirements since it’s all dealing with legal jurisdictions other than my own), but i wanted to make explicit that someone who knows the legalities and is willing 
 to put 
in the work can reach out to me.

We can start a separate thread about discovery, using a separate PEP
to formalize it.

This could be as simply as having a flag saying "use the download URL
as index and offer this to the user trying to find the package distribution
files".

> The other major difference is that I’ve shortened the time schedule from 6 months to 3 months. Given that authors are either going to upload their projects to PyPI or not and there is no longer a need to setup an external index I think a shorter time schedule is fine, especially since they will be given a script they can run that will spider their projects for any installable files and upload them to PyPI for them in a quick one shot deal that would require very little effort for them.

It would be good to have both PEP 470 and the discovery PEP available
at the same time.

> Everything else in the PEP is basically the same except for rewordings.
> 
> I do need a BDFL Delegate for this PEP, Richard does not have the time to do it and the other logical candidate for a PyPI centric PEP is myself, but I don’t feel it’s appropriate to BDFL Delegate my own PEP.
> 
> You can see the PEP online at https://www.python.org/dev/peps/pep-0470/ (make sure it’s updated and you see the one that has Aug 26 2015 in it’s Post History).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 27 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-08-19: Released mxODBC 3.3.5 ...             http://egenix.com/go82

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/