[Catalog-sig] V4 Pre-PEP: transition to release-file hosting on PYPI

[Richard Jones]

Some suggested edits; I'm otherwise quite happy with the current draft.

On 15 March 2013 02:29, holger krekel <holger at merlinux.eu> wrote:
> History and motivations for external hosting

Could we please have a reference to the Package Index "API"* here?


> Today, most packages released on PyPI host their release files on
> PyPI, but a small percentage (XXX need updated data) rely on
> external hosting.

The above should probably be re-worded since "rely" is loaded and we
don't necessarily know the motivation for projects using external
links. The important numbers though are:

projects with any external only links: 2581
projects with only external only links: 1332
total projects: 29117

Whether the projects with links that also have hosted files (ie. the
1249 project difference between those numbers) *rely* on us retaining
the external links facility is unknown.


> Hosting modes
> -------------
>
> The foundation of the first transition phase is the introduction of
> three "modes" of PyPI hosting for a package, affecting which links are
> generated for the ``simple/`` index.  These modes are implemented
> without requiring changes to installation tools via changes to the
> algorithm for generating the machine-readable ``simple/`` index.
>
> The modes are:
>
> - ``pypi-scrape-crawl``: no change from the current situation of
>   generating machine-readable links for installation tools, as
>   outlined in the history_.
>
> - ``pypi-scrape``: for a package in this mode, links to be added to
>   the ``simple/`` index are still scraped from package
>   metadata. However, the "Home-page" and "Download-url" links are
>   given ``rel=ext-homepage`` and ``rel=ext-download`` attributes
>   instead of ``rel=homepage`` and ``rel=download``. The effect of this
>   (with no change in installation tools necessary) is that these links
>   will not be followed and scraped for further candidate links by present-day
>   installation tools: only installable files directly hosted from PYPI or
>   linked directly from PyPI metadata will be considered for installation.
>   Installation tools MAY evolve to offer an option to use the new
>   rel-attribution to crawl external pages but MUST NOT default to it.

I'd just like to confirm that the rel="download" / rel="ext-download"
switch will not affect the installability of distribution downloads
linked directly by download_url.


> - ``pypi-explicit``: for a package in this mode, only links to release
>   files uploaded to PyPI, and external links to release files
>   explicitly nominated by the package owner (via a new interface
>   exposed by PyPI) will be added to the ``simple/`` index.

The bracketed bit there needs to be emphasised (ie. not just a
bracketed afterthought) as it changes the current packaging user
experience considerably for those who wish to remain externally
hosting files.



     Richard

* http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api