[Distutils] vetting, signing, verification of release files

[Trishank Karthik Kuppusamy]

On 07/17/2013 04:50 PM, Nick Coghlan wrote:
>
> On 17 Jul 2013 18:17, "holger krekel" <holger at merlinux.eu 
> <mailto:holger at merlinux.eu>> wrote:
> >
> > On Wed, Jul 17, 2013 at 07:48 +0000, Vinay Sajip wrote:
> > > holger krekel <holger <at> merlinux.eu <http://merlinux.eu>> writes:
> > >
> > > > about existing schemes/efforts.  I guess most Linux distros do 
> it already
> > > > so if nothing comes up here PyPI-specific (what is the status of 
> TUF, btw?)
> > > > i am going to look into the distro's working models.
> > >
> > > ISTM it works for distros because they're the central authority 
> guaranteeing
> > > the provenance of the software in their repos. It's harder with 
> PyPI because
> > > it's not a central authority curating the content. Perhaps 
> something like a
> > > web of trust would be needed.
> >
> > I am thinking about curating release files _after_ publishing and
> > then configuring install activities to require "signed-off" release 
> files.
> > Basically giving companies and devops the possibility to organise their
> > vetting processes and collaborate, without requiring PyPI to change 
> first.
> > This certainly involves the question of trust but if nothing else an 
> entity
> > can at least trust its own signatures :)
>
> Note that Linux distros don't trust each other's keys and nor do app 
> stores trust other. Secure collaborative vetting of software is an 
> Unsolved Problem. The Update Framework provides a solid technical 
> basis for such collaboration, but even it doesn't solve the 
> fundamental trust issues. Those issues are why we still rely on the CA 
> model for SSL, despite its serious flaws: nobody has come up with 
> anything else that scales effectively.
>
> The use of JSON for metadata 2.0 is enough to make it TUF friendly, 
> but there are significant key management issues to be addressed before 
> TUF could be used on PyPI itself. That's no reason to avoid 
> experimenting with private TUF enabled PyPI servers, though - a 
> private server alleviates most of the ugly key management problems.
>
>

Thank you, Nick. Indeed, we think that TUF (designed in collaboration 
with some of chief designers of the Tor project) offers a secure yet 
usable way to address many classes of attacks on package managers, many 
previously left unconsidered in the Linux distribution community until 
we pointed it out to them, at which point they adopted our security 
proposals (https://isis.poly.edu/~jcappos/papers/cappos_mirror_ccs_08.pdf).

We are delighted to see that JSON is being used for PyPI metadata 2.0, 
which would certainly lend itself very easily for integration with TUF. 
Speaking of which, let me answer some questions about the current status 
of PyPI and pip over TUF.

TLDR: We now have a pretty good scheme balancing key management with 
security for PyPI. At the time of writing, I have an almost identical 
version of pip ready anytime to read metadata off a TUF-secured PyPI 
mirror. There is just one thing left to do: I need to just compress the 
metadata as much as possible (a problem common to all package managers). 
I expect this to be done in the next two weeks, by which time we should 
have a slightly modified version of pip which would securely download 
packages from an up-to-date TUF-secured PyPI mirror.

(Aside: let me say that we are discussing all things related to PyPI, 
pip and TUF on the TUF mailing list 
(https://groups.google.com/forum/?fromgroups#!forum/theupdateframework). 
I welcome you to join our mailing list so that we can continue the 
discussion. I did not want to incessantly copy our discussions to the 
DistUtils mailing list because I was not sure whether it would be always 
relevant to the DistUtils SIG which is already busy with a number of 
other projects. In retrospect, perhaps I should have summarized our 
findings every now and then on this list, because I can understand that 
it looks to some people as though we have been silent, when in fact that 
was not the case.)

To very briefly summarize our status without going into tangential details:

1. We previously found and reported on this mailing list that if we 
naively assigned a key to every PyPI project, then the metadata would 
not scale. We would have security with little usability. This looks like 
an insoluble key management problem, but we think we have a pretty good 
solution.
2. The solution is briefly this: we now propose just two targets roles 
for all PyPI files.
2.1. The first role --- called the "unstable" targets role --- will have 
completely online keys (meaning that it can kept on the server for 
automated release purposes). The unstable role will sign for all PyPI 
files being added, updated or deleted without question. The metadata for 
this role will change all the time.
2.2. The second role --- called the "stable" targets role --- will have 
completely offline keys (meaning that keys are kept as securely as 
possible and only used with manual human intervention). The stable role 
will sign for only the PyPI files which have vetted and deemed 
trustworthy. The metadata for this role is expected to change much less 
frequently than the unstable role.

Okay, sounds too abstract to some. What does this mean in practice? We 
want to make key management simple. Preferably, as Nick Coghlan and 
others have proposed before, we would want PyPI to initially, at least, 
sign for all packages, because managing keys for every single project 
right off the bat is potentially painful. Therefore, with that view in 
mind --- which is to first accommodate PyPI signing for packages, and 
gradually allowing projects to sign for their own packages --- we then 
consider what our proposal above would do.

Firstly, it would make key management so much simpler. There is a 
sufficient number of offline keys used to sign metadata for a valuable 
and trustworthy set of packages (done only every now and then), and an 
online key used to make continuous release of PyPI packages possible 
(done all the time).

1. Now suppose that the top-level targets role says: when you download a 
package, you must first always ask the stable role about it. If it has 
something to say about it, then use that information (and just ignore 
the unstable role). Otherwise, ask the unstable role about it.
2. Fine, what about that? Now suppose that the both the stable and 
unstable roles have signed for some very popular package called FooBar 
2.0. Suppose further that attackers have broken into the TUF-secured 
PyPI repository. Oh, they can't find the keys to the stable role, so 
they can't mess with the stable role metadata without getting caught, 
but since the unstable keys are online, they could make it sign for 
malicious versions of the FooBar 2.0 package.
3. But no problem there! Since we have instructed that the stable role 
must always be consulted first, then valid metadata about the intended, 
trusted FooBar 2.0 package cannot be modified (not without getting all 
the human owners of the keys to collude). The unstable role may be 
tampered with to offer bogus metadata, but the security impact will be 
limited with *prior* metadata about packages in the way-harder-to-attack 
stable role.

More details, should you be interested, are available here: 
https://groups.google.com/forum/?fromgroups#!topic/theupdateframework/pocW9bacwuc

I hope that answers a number of questions. Let us know if you have more 
questions, and I think I can safely conclude that I can start discussing 
TUF on this mailing list again!

PS: Pardon any delay in my response in the next couple of days, as I 
will be flying for a day or so to New York in approximately 24 hours.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130718/8d128e79/attachment-0001.html>