[Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

[Donald Stufft]

On May 8, 2014, at 10:36 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Donald Stufft <donald at stufft.io> wrote:
>> There is support for trusted externally hosted packages, you put the URL in
>> PyPI and include a hash in the fragment like so:
>> 
>> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56
> 
> That is exactly the mode I was using until today.  This mode produced the
> subject's warning message.
> 
> Today I've switched to manual install mode with manual sha256sum verification
> which is *far* safer than anything you get via pip right now.

It is not safer in any meaingful way.

If someone is in a position to compromise the integrity of PyPI's TLS, they
can replace the hash on that page with something else. Now you've attempted to
work around this by telling people to go look up the release announcement
hash. However if someone can compromise the integrity of PyPI's TLS, they can
also compromise the integrity of https://mail.python.org/, or GMane, or any
other TLS based website[1].

All of that assumes that the end user is going to bother to verify the hash
*at all* which almost none of them will and they'll just check the http url
into their requirements.txt file and be downloading things over HTTP and
be vulnerable to arbitrary code execution via MITM.

> 
> 
>> [2] For the definition of safe that PyPI/pip operate under, which is that the
>>    author of a package is assumed to be trusted by the person electing to
>>    download their package.
> 
> No, there are other holes, which you have conceded in your previous mail.

The presence of other holes is not a useful argument to avoid closing a hole.
We're working to close all of them, and that ends up meaning we close one at
a time.

> 
> 
>> I don't think the warning is FUD, and it doesn't mention anything security
>> related at all. The exact text of the warning is in the subject of the email
>> here:
>> 
>>    cdecimal an externally hosted file and may be unreliable
>> 
>> Which is true as far as I can tell, it is externally hosted, and it may be
>> unreliable[1]. If there is a better wording for that I?m happy to have it and
>> will gladly commit it myself to pip.
> 
> Do you honestly not see a difference between the cited warning and the
> *intended* warning "the server's availability may be unreliable”?

Do I? No I don’t. However I’ve since adjusted the message based on
R David Murray’s feedback to make sure it specifically says that access
may be unreliable instead of just that the package itself may be unreliable.

> 
> Even the latter is FUD or a truism (it applies to any server).

No, because the use of an external host *adds* additional unreliability. If
PyPI is down, then all packages are down, including ones that host externally.
If the cdecimal server is down, then that one specific package is unavailable.

It is impossible to do anything but reduce the overall availability by adding
additional SPOFs.

> 
> The real question is:  Why is there a warning if the person running pip
> has explicitly allowed external packages?
> 

Why is there a warning? Originally that warning was there because the first
part of the transition to this "mode" of defaults was to give an option to
disable externally hosted files, but leave it on by default. In this phase
we gave this warning to tell the people who just leave things to their default
about this file.

Should the warning itself still exist? I don't know, but a better avenue for
asking for a change in pip is via our issue tracker instead of whining on
python-dev.

> Stefan Krah
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/2af61db6/attachment-0001.sig>