[Distutils] Deprecating little used file types/extensions on PyPI?

[Wes Turner]

On Tuesday, August 23, 2016, Donald Stufft <donald at stufft.io
<javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:

>
>
> It’s hard to get per project numbers for these (or at least, it takes a
> more
> complex query than I can manage with my head here). However, I think it’s
> pretty telling that when you start looking at other formats, not only is
> the
> primary consumer tools that just indiscriminately download everything from
> PyPI,
> but almost *all* of the consumers of those files are tools that just
> indiscriminately download everything. Unless there are users of those
> mirrors who
> follow vastly different usage patterns than what we see on PyPI itself,
> the primary
> purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume
> disk space
> and bandwidth via the mirroring infrastructure.


Other ways to reduce mirroring bandwidth:

Caching proxies:

- devpi (proxy cache only what is needed)
  http://doc.devpi.net/latest/quickstart-pypimirror.html
- "pypi proxy" https://www.google.com/search?q=pypiproxy

- nginx caching proxy
  - https://gist.github.com/dctrwatson/5785638
    - this would need to be updated for the pypi.org warehouse routes
    - it may be worth linking to these in the/some mirroring infrastructure
docs

Pip --offline switch

- Internet was down the other day and i knew the packages were locally
cached but I couldn't figure out how to make pip work w/ local offline
dependency resolution

More efficient mirroring implementations:

- rsync
- bup
- DRPM


... delta downloads would be more efficient here:
https://github.com/pypa/warehouse/issues/347
"Add API endpoint to get latest version of all projects"




> I’d also like to note, that the numbers above are conservative on what they
> consider to be a “mirroring client”. For instance, devpi used to use the
> default
> requests user-agent, and we see downloads via the requests user agent, but
> did not
> count them as mirroring clients because it could be some other script
> doing the
> downloading.
>
> —
> Donald Stufft
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/ca53e2e3/attachment.html>