[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Donald Stufft]

On May 11, 2014, at 6:49 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 12 May 2014 08:12, "Donald Stufft" <donald at stufft.io> wrote:
> > On May 11, 2014, at 6:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> So, re-reading that, my preference is for option 3: keeping the global allow-all-external flag, but renaming it as something like "allow-all-verifiable-external". It's only dropping that flag entirely or making it mean "allow all unverifiable" that would mean moving away from the previously agreed approach in the PEP.
> >>
> >> There's no requirement in the PEP for a per-package flag to accept verifiable downloads, so making allow-external mean the same thing as allow-unverifiable isn't a problem from that perspective. The PEP also doesn't mandate particular option names.
> >
> >
> > The problem is, if you’re reading the documentation it looks like
> > --allow-all-verifiable-external is the option you want. People fundamentally
> > don’t grasp the difference between safe and unsafe external hosting. I've tried
> > to explain it over and over and over and the most common reaction is confusion.
> > You read the documentation and you’re told “We don’t allow externally hosted
> > things by default, you can use —allow-external <package> to allow it for a
> > particular package which will include unsafely hosted ones, or you can use
> > --allow-all-verifiable-external to allow all safely hosted ones.
> 
> I agree we'd never recommend the global flag, as it probably won't work. This is really about managing the removal process in a way that makes it clear that we're acting as responsible stewards in the spirit of PEP 438, rather than being cavalier in removing options.
> 
> In that vein, deprecation of the global flag would be more appropriate than removal (I'm assuming you don't want to change it to globally opt in to unverifiable external downloads).
> 
> Specifically, what I would recommend is:
> 
> 1. Remove any references to the global flag from other documentation and error messages - always recommend the use of per-package flags to enable external downloads.
> 2. Deprecate the global flag in 1.6
> 3. Remove it in 1.7 unless there is a marked increase in the proportion of verifiable vs unverifiable external hosting on PyPI.
> 
> Adding a more explicit alias in step 2 would also be reasonable (since the meaning of the base "allow external" will be changing). Either way, the docs for the global option should acquire a caveat:
> 
> "Note: The vast majority of links from PyPI to externally hosted packages are missing the hash information needed for verification, so this option won't actually allow downloading of most externally hosted packages - to handle such cases, use "allow-external" with the specific package name"
> 
> Cheers,
> Nick.

So, this is essentially my current PR except instead of removing the
--allow-all-external flag in 1.6 it's deprecated and removed in 1.7. I was on
the fence about removing vs deprecating the --allow-all-external flag. The only
reason I chose to remove it was I didn't feel very good about making it match
the semantics of --allow-external (external safe and unsafe) but the naming of
it made it sound like it was just the global option.

I'm not completely against that. I think it's choosing between three bad
options, removing --allow-all-external, making it not match --allow-external,
or making it a global do an unsafe thing flag.

However before I go further on that I want to dig more into the impact of these
things. It dawned on me earlier today that the way I was categorizing things
in my earlier number crunching was making it unreasonably hard to actually
divine any sort of meaning out of those numbers. I'm currently in the process
of crawling all of PyPI again*, after I have those new numbers I'll have a
better sense of things and I think a better forward plan can be made.

* This currently takes around 3 hours to do a full scan, 2.5 hours of that is
  due to the external links we have to process.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140511/33831051/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140511/33831051/attachment-0001.sig>