[Distutils] a plea for backward-compatibility / smooth transitions (was: Re: Migrating Hashes from MD5 to SHA256)

[Donald Stufft]

On Jul 30, 2013, at 3:56 AM, holger krekel <holger at merlinux.eu> wrote:

>> Yes. I've been using sha256 on simple.crate.io for over a year and
>> zero people have ever stated it didn't work for them. This also fits
>> in with my knowledge of how setuptools and pip works. I know
>> zc.buildout less well but to my knowledge they simple allow setuptools
>> to handle the downloading.
> 
> Sounds good.  Maybe "secondary" tools could get problems, though.
> I know for sure that devpi-server might stumble but i can fix that.
> Also i remember there were tools that memorized MD5 hashes in requirements
> files etc.

The change was nixed but it wasn't about removing the ability for pip and
such to use MD5s. Merely what PyPI serves.

>> Registered externals must register with a md5 hash, scraped links and
>> download urls etc do not require it because they are indirectly added.
>> There is no verification by PyPI that the given hash matches the
>> package at the end of the url.
> 
> Hum, can we allow submitting multiple hashes?  Are there tools already 
> that help with registering externals?

Not easily and in a backwards compatible way.

>> MD5 is currently broken for collision resistance. This means that an
>> author can generate two packages that hash to the same thing. Once
>> package might be benign and one might be malicious. Given those two
>> packages people using the md5 hashes will not be able to differentiate
>> between the benign and the malicous package.
> 
> I think we should not pretend that PyPI has (by itself) any safety belts 
> against malicious authors.  There are numerous ways for malicious authors
> to do evil if they choose to.  The potential ability to fake a package
> using a collision attack merely adds another way.

Correct, which is why I tried to be very specific about the types of attacks :)

> 
> Do you know, btw, if TUF is going to help with any of what we are discussing
> here? (I am again a bit lost as to the roadmap wrt to TUF - is there
> something?)

TUF would provide protections against a pre-image attack yes. However it has
it's own problems and is still likely a ways out (if we use it).

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130730/170e9c1c/attachment-0001.pgp>