[Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting

[Paul Moore]

On 7 October 2014 11:09, holger krekel <holger at merlinux.eu> wrote:
> Well, the main benefit of PEP438 was that it removed random crawling for
> some 90% of the packages on the package index, speeding up and making
> installs more reliable.  And it did that without breaking backward
> compatibility.

The setuptools index page is 1.4MB in size. Most of that can be
ignored, but it still has to be downloaded and parsed. Whether the
data that setuptools includes in its long_description is useful is
arguable, but irrelevant - the fact is that as things stand, it is
there and it causes issues.

PEP 470 would result in all of the unneeded entries in the simple
index for setuptools being removed, which avoids the need for client
tools (and I'm not talking just about pip here, but also about one-off
scripts, which is the sort of thing I write a lot) to trawl through
all of that data. And it does so without the setuptools project having
to change how it writes its PyPI page (i.e., the project
long_description).

Arguably, that's equally a way of avoiding breaking backward compatibility...

> The second could be done without breakage alltogether i think:  at one
> time all external urls are auto-registered as external indexes
> and they are presented on the simple page with some meta information
> that does not confuse older pips/easy_installs.  Newer pips/easy_installs
> can then provide nice error messages.  Older pips can continue to use
> the PEP438 options.  And easy install can continue to work.

Setuptools has 255 internal links to files hosted on PyPI. And about
11,000 other links. (I just checked that 3 times, as I couldn't
believe it, but it *seems* to be right :-(). Removing duplicates, 337
unique links. Are you suggesting pip presents all of those as possible
external indexes?

I'm sure you can argue that setuptools has (badly!) misused the
link-handling support in PyPI. And that it's a one-off special case.
But how do we document to projects that they shouldn't do things like
this? How do we even define what "things like this" are? Don't include
links in your project description unless they are external indexes?

Paul.