[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Paul Moore]

On 9 May 2014 15:05, Donald Stufft <donald at stufft.io> wrote:
> So originally in order to get pip to consider external hosted files at all
> you had to have —allow[-all]-external.

Well, *originally* you needed to do nothing. It was only PEP 438 that
made it necessary to do any of this.

> On top of that if you wanted to install something that was unverifiable
> you would have to add —allow-unverifiable.
>
> This meant to install something that was “unsafe” you had to do:
>
> $ pip install —allow-external foo —allow-unverifiable foo foo
>
> This has since been changed so that —allow-unverifiable implies
> —allow-external so that all that would be required is:
>
> $ pip install —allow-unverifiable foo foo
>

None of the above is relevant to my concerns, because it is relating
to *unsafe* packages.

> So that’s the state of the flags, as to the actual messages that people
> get:
>
> pip does not attempt to discover (most) unsafe links if it knows it isn’t
> going to use them. It does this as a (substantial) performance
> improvement. However the side effect of this is that we don’t actually
> know ahead of time if there are any unverifiable files so at best what
> we can do is saying that maybe there are some files that can be
> discovered if you add a flag.

I can't make sense of whether this is relevant. You are still talking
about unsafe links, which I have repeatedly said I have no issue with.
How is this relevant to handling of *safe* external links? I get a
feeling that you're trying to explain something that is relevant ("a
side effect of this is...") but I really don't see what that is. Can
you explain again, but without any reference to processing of unsafe
links? That might help me to understand your point.

> Typically what people do is they add the —allow-external flag, find
> it doesn’t work then add the —allow-unverifiable flag. The —allow-external
> flag rarely actually helps them because very few projects are
> safely hosted externally.

Agreed. So my point is that the --allow-external flag very rarely
helps anyone, so make its behaviour the default so they don't need to
worry. If the flag does help some people, we need to know who it
helps, so we can balance the benefits against the cost to all the
people who get confused over whether they want --allow-unverifiable or
--allow-external, and come up with a less error-prone solution.

>> I see your point but honestly I'd expect such a developer to simply
>> put allow-all-external=true in his pip.ini once, probably so long ago
>> that he's forgotten, so there's no gain. If he actually cares enough
>> to track externally hosted files, adding a comment to requirements.txt
>> would be just as good, and having something like ```pip install
>> --list-external``` to show him what was externally hosted helps him
>> get the information he needs.
>
> I’m not sure anyone has ever mentioned to me that they’ve done this,
> I do have people who have mentioned to me that they have
> added the —allow-external foo to their requirements.txt. Obviously
> not scientific or complete but just what I’ve been told.

I'm pretty sure I'll do precisely that the first time I hit the issue.
I may well do it pre-emptively at some point. Whether that counts as
me being a data point or just me sulking, I'll leave to others to
decide :-) But I will say that 99% of my usage is manual entry of
requirements on the command line, and not requirements files, which
may affect my views ("just add it to requirements.txt and you're done"
isn't an answer for me).

Paul