[Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

[Donald Stufft]

On May 9, 2014, at 5:01 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 9 May 2014 05:34, Donald Stufft <donald at stufft.io> wrote:
>> On May 8, 2014, at 5:22 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>>>> Socially, this change does not seem to be having the effect of
>>>> persuading more package developers to host on PyPI. The stick doesn't
>>>> appear to have worked, maybe we should be trying to find a carrot?
>>> 
>>> Do you have any data to point to that says it hasn’t worked? Just to see
>>> what impact it has had, I’m running my scripts again that I ran a year
>>> ago to see what has changed, already I can see they are processing
>>> MUCH faster than last year.
>> 
>> The data has finished processing, it represents a time diff of approximately
>> one year. The pip release that caused all of this was released about 4-5 months
>> ago.
>> 
>> Overall PyPI has seen a 50% growth in installable projects in that time. If the
>> change would have had no effect we'd expect to see a ~50% increase across the
>> board. However what we've seen is a a 60% (+10% of expected) increase in
>> projects that can only be installed from PyPI and a 12% decrease in projects
>> that have any unsafe files (-62% of expected).
> 
> Donald,
> Thanks for taking the time to get those figures. It does appear that
> there are less cases that would be affected than the number of
> complaints would imply.

Of course, I don’t like making claims without backing them up if I can :)

> 
> The only concern I have about this type of analysis is that it doesn't
> "weight" projects. It may be (and again, I have no data to back this
> up) that the projects that are affected detrimentally by this change
> are unusually popular or otherwise significant. There's obviously no
> way to assess this sensibly other than by making a judgement on the
> level of complaints.

Yea, I don’t have a good way to weight those projects in any way. Normally
I could get some sort of estimate by looking at the download numbers from
PyPI but well ;)

For the record, here’s the list of projects that are hosted *only* safely externally
or that have *any* safely externally hosted files:

https://gist.github.com/dstufft/1b16c305f97fff6cef2f

Most of these don’t stand out to me at all. The only ones that do are:

* pyOpenSSL which has one older release that is hosted that way
* argparse which has the latest release hosted this way but has
  older releases hosted on PyPI
* new relic which only hosts older releases externally
* beautifulsoup4 which hosts things safely externally *and* on PyPI
* Paste which has one “external” thing which is actually only external
  because it used a cheeseshop.python.org link instead of a pypi.python.org
  link.
* ipython which has one older release hosted safely externally but the
  latest is on PyPI
* netifaces which has one older release hosted safely externally but the
  rest are on PyPI

> 
> But arguing numbers was never my intention here, so let's just say
> that I concede that the change has had a positive effect, which is
> great.
> Paul

I didn’t mean to try to imply that it was :) I just wanted to make sure that
*my* claims were true, or if they weren’t I wanted to be able to say that
I was wrong. Since I had the numbers computed already it didn’t make
any sense not to share them here.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/ba0535d3/attachment.sig>