[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Donald Stufft]

On May 9, 2014, at 11:26 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 9 May 2014 15:05, Donald Stufft <donald at stufft.io> wrote:
>> So originally in order to get pip to consider external hosted files at all
>> you had to have —allow[-all]-external.
> 
> Well, *originally* you needed to do nothing. It was only PEP 438 that
> made it necessary to do any of this.

Orginially in the implementation of PEP438. It was part of an explanation
about why people get confused between —allow-external and —allow-unverifiable
and mentioning a change that I made to pip to try and make that simpler.

> 
>> On top of that if you wanted to install something that was unverifiable
>> you would have to add —allow-unverifiable.
>> 
>> This meant to install something that was “unsafe” you had to do:
>> 
>> $ pip install —allow-external foo —allow-unverifiable foo foo
>> 
>> This has since been changed so that —allow-unverifiable implies
>> —allow-external so that all that would be required is:
>> 
>> $ pip install —allow-unverifiable foo foo
>> 
> 
> None of the above is relevant to my concerns, because it is relating
> to *unsafe* packages.

This was part of an explanation of why they get confused between the
two options again.

> 
>> So that’s the state of the flags, as to the actual messages that people
>> get:
>> 
>> pip does not attempt to discover (most) unsafe links if it knows it isn’t
>> going to use them. It does this as a (substantial) performance
>> improvement. However the side effect of this is that we don’t actually
>> know ahead of time if there are any unverifiable files so at best what
>> we can do is saying that maybe there are some files that can be
>> discovered if you add a flag.
> 
> I can't make sense of whether this is relevant. You are still talking
> about unsafe links, which I have repeatedly said I have no issue with.
> How is this relevant to handling of *safe* external links? I get a
> feeling that you're trying to explain something that is relevant ("a
> side effect of this is...") but I really don't see what that is. Can
> you explain again, but without any reference to processing of unsafe
> links? That might help me to understand your point.

I’m trying to explain why people tend to do the “add a flag, that fails,
then add a second flag” and that reason is because generally what
they want is —allow-unverifiable but our messages are kind of crappy
and tricks them into thinking —allow-external is what they want.

> 
>> Typically what people do is they add the —allow-external flag, find
>> it doesn’t work then add the —allow-unverifiable flag. The —allow-external
>> flag rarely actually helps them because very few projects are
>> safely hosted externally.
> 
> Agreed. So my point is that the --allow-external flag very rarely
> helps anyone, so make its behaviour the default so they don't need to
> worry. If the flag does help some people, we need to know who it
> helps, so we can balance the benefits against the cost to all the
> people who get confused over whether they want --allow-unverifiable or
> --allow-external, and come up with a less error-prone solution.

It rarely helps anyone to enable it, but the *choice* to enable it is important
because people using pip rarely expect that it’s going to touch hosts
other than PyPI and generally are quite confused when it does.

> 
>>> I see your point but honestly I'd expect such a developer to simply
>>> put allow-all-external=true in his pip.ini once, probably so long ago
>>> that he's forgotten, so there's no gain. If he actually cares enough
>>> to track externally hosted files, adding a comment to requirements.txt
>>> would be just as good, and having something like ```pip install
>>> --list-external``` to show him what was externally hosted helps him
>>> get the information he needs.
>> 
>> I’m not sure anyone has ever mentioned to me that they’ve done this,
>> I do have people who have mentioned to me that they have
>> added the —allow-external foo to their requirements.txt. Obviously
>> not scientific or complete but just what I’ve been told.
> 
> I'm pretty sure I'll do precisely that the first time I hit the issue.
> I may well do it pre-emptively at some point. Whether that counts as
> me being a data point or just me sulking, I'll leave to others to
> decide :-) But I will say that 99% of my usage is manual entry of
> requirements on the command line, and not requirements files, which
> may affect my views ("just add it to requirements.txt and you're done"
> isn't an answer for me).
> 
> Paul

You’re unlikely to ever encounter an issue where adding —allow-external
actually helps you, but adding it does make it more likely you’ll be hurt.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140509/75438f1a/attachment-0001.sig>