[Catalog-sig] Deprecation of External Urls, Statistics

[Jesse Noller]

As long as external URLs eventually are completely removed I'm okay with caching things

On Mar 8, 2013, at 6:49 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:

> On 08.03.2013 02:40, Donald Stufft wrote:
>> So I updated my script (had to remove eventlet) and I believe it's now accurate. The total time was ~54 hours so this is hardly scientific but it should give a good idea what sort of impact we are talking about.
>> 
>> This is a list of versions that pip's PackageFinder (what it uses to locate packages to install) could find that were not available on PyPI.
>> 
>> The results and script is available at: https://gist.github.com/dstufft/5088915
>> 
>> Some statistics:
>> 
>>    Projects affected (with dev): 2269
>>    Versions affected (with dev): 8006
>> 
>>    Projects affected (without dev): 1880
>>    Versions affected (without dev): 7586
>> 
>> These numbers are if all external urls were immediately removed from PyPI, so this would be the total affected. This does not test if the actual package is installable, just if pip is able to locate an url that it thinks represents a version for that project.
> 
> Thanks for running the test.
> 
> About 10% of all packages. The numbers are already impressive,
> but if you factor in the popularity of some of those
> packages, the situation becomes worse.
> 
> I'm beginning to wonder whether caching the external link content
> on the PyPI CDN wouldn't be a better idea.
> 
> We'd have to make that legally waterproof and also have an opt-out
> mechanism, but it would get us from here to there a lot faster.
> 
> Together with the added hash tag on the download file URLs (*),
> this would solve the availability and the security aspects.
> Instead of deprecating external links altogether, we could then
> deprecate non-compliant download links and get an overall
> very flexible system for Python package distribution.
> 
> (*) Yes, I know, I still have to deliver the updated proposal -
> been working on getting our indexes ready to serve as example :-)
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, Mar 07 2013)
>>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig