[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Donald Stufft]

On May 9, 2014, at 6:16 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> So there's an ongoing debate over pip's behaviour around disallowing
> external hosting by default (see thread "pip: cdecimal an externally
> hosted file and may be unreliable" over on python-dev for the latest
> round).
> 
> It appears that the reason for disallowing external hosting (as
> opposed to unverifiable downloads) is purely about reliability - we
> can't be sure that an external host provides the same level of uptime
> as PyPI[1]. Given that, it seems to me that the situation is, for an
> externally hosted package foo:
> 
>    `pip install foo` - fails immediately, 100% of the time
>    `pip install --allow-external foo foo` - works in all but a few
> cases where foo's host is down[2]
> 
> I cannot understand how guaranteed failure is ever better than
> "occasional but rare" failure.
> 
> For situations where it is critical to minimise the risk of an
> external host outage causing a deployment to fail, the only answer is
> to not use foo, or to host foo on your own private index. In both
> cases, all you need is to know that foo is externally hosted to do
> that - you certainly don't need pip to fail.
> 
> As a concrete proposal:
> 
> 1. Remove --allow-external/--allow-all-external and make it the
> default behaviour.
> 2. Add a new command to pip, maybe something like `pip check-external`
> which checks a set of reuirements, and reports the requirements that
> are externally hosted and which hosts they rely on. That gives users
> who need 100% reliability the information they need to implement the
> appropriate solution. Without causing pain for users who don't.
> 
> Note that the above is based on the fact[3] that *there are no
> security risks to --allow-external*. I am not arguing for a reduction
> in security, or a change to any sort of threat model.
> 
> Comments?
> 
> Paul
> 
> [1] Donald explicitly stated that this is the case in the earlier
> thread (https://mail.python.org/pipermail/python-dev/2014-May/134454.html).
> I think Nick confirmed this (although I don't have a reference to
> hand). If it's not true then we need to be a lot clearer and more
> explicit about *why* ignoring external hosting by default is needed,
> because it seems nobody knows :-(
> [2] BTW, the syntax of `--allow-external` is hideous, but that's a
> side issue I don't want to debate right now.
> [3] See the first note.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

I replied on python-dev already, but I’m still heavily -1.

This isn’t actually going to help hardly anyone since almost no packages are
hosted safely externally. I think the only practical affect is that the latest version
of argparse will be installable.

I think if we actually care about end user confusion the actual thing to do is remove
the concept of safe externally hosted all together and just make it Hosted on
PyPI Y/N? If it is install it if it isn’t require a per package flag.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140509/59bff09e/attachment.sig>