[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Paul Moore]

On 9 May 2014 13:17, Donald Stufft <donald at stufft.io> wrote:
> I replied on python-dev already, but I’m still heavily -1.

Yeah, I have no idea if the discussion will migrate here or not, but I tried :-)

> This isn’t actually going to help hardly anyone since almost no packages are
> hosted safely externally. I think the only practical affect is that the latest version
> of argparse will be installable.

The same argument could be made about implementing the external
hosting restriction in the first place. From the figures you posted on
python-dev:

> * We've gone from 86% of projects being installable from PyPI to 92%.
> * We've gone from 5% of projects being only unsafely installable to 3%
> * We've gone from 14% of projects having any files unsafe to install to 8%
> * We've gone from 0.004% of projects being safely hosted externally to 0.2%

There's a lot about unsafe installs here, which is a distraction, but
it seems to me that safe external hosting has gone from 0.004% to
0.2%. Maybe a substantial chunk of this is projects making external
downloads safe, so let's assume that no new projects have been added
that are externally hosted (safely). So this whole feature only
affects 0.2% of PyPI at best.

I wish this feature had never been added, because there is *no way*
that the benefits (between 0.004% and 0.2% of PyPI) justify the amount
of time discussing it, dealing with bug reports about it, etc. Please
again note that this is SOLELY the default rejection of SAFE external
hosting. Figures and arguments about unsafe hosting aren't relevant
here.

> I think if we actually care about end user confusion the actual thing to do is remove
> the concept of safe externally hosted all together and just make it Hosted on
> PyPI Y/N? If it is install it if it isn’t require a per package flag.

I am strongly -1 on any such suggestion. The concept of *unsafe*
hosting is crucially important. The fact that it's external is
peripheral, simply because we ensure that PyPI is safe. We already
have --allow-unverified and a default of not allowing unverified
downloads. I'm arguing that we don't need a second set of flags, and
that the confusion is that users see the 2 sets of flags and assume
that both conditions are somehow equally bad.

BTW, you didn't respond to my point:

Previously,
    `pip install foo` - works in all but a few cases where foo's host is down

Now
    `pip install foo` - fails immediately, 100% of the time
    `pip install --allow-external foo foo` - works in all but a few
cases where foo's host is down

How is the second better for the pip user than the first? What did the
user learn that they can actually use to improve things for themselves
in any way, assuming that they already knew foo is externally hosted?

I'll give up the fight at this point. I don't know this part of the
pip code well enough to offer a patch implementing my suggestion, and
in any case I think that duelling patches is a very unhealthy thing
for a project. So as you did the work, I'll accept your view. But
could I ask that in future debates like this, if anyone suggests that
the pip developers are mandating the current behaviour, that mistake
is immediately corrected to point out that it's the PEP 438 authors
who are in fact doing this (and that pip is merely following the
requirements in that PEP)? I will certainly do this myself.

Paul