[Distutils] changelog / CDN inconsistency (was: Re: Good news everyone, PyPI is behind a CDN)

[holger krekel]

On Tue, May 28, 2013 at 11:04 +0200, Christian Theune wrote:
> On 27. May2013, at 10:41 PM, Donald Stufft <donald at stufft.io> wrote:
> > Just to assure folks. I do consider Mirroring a first class citizen and an important feature.
> 
> Thanks for that acknowledgement. Lets sort out what to do now - this is becoming urgent for me as the author of the currently recommended mirroring tool for public mirrors and as an operator of a mirror that is being relied upon.
> 
> I agree with Holgers points.
> 
> I don't think the mirroring is completely backwards right now. I agree there's been an incomplete PEP that's been hanging around too long. 
> 
> My current client implementation is pretty simple and has had reliable semantics until now.
> 
> A couple of things I noticed in the discussion that I'd like to point out:
> 
> - We mirror simple pages because the PEP requires us to - this is part of the existing validation approach. I can drop that to get mirrors not to rely on simple pages from the CDN but then authentication of the simple pages will be broken.
> 
> - Release files are replaced all the time.
> 
> The semantics that I like to keep with the mirrors is this:
> 
> When I get a changelog for serial X and I start copying simple pages and files then I (as a mirror) promise my clients that I have incorporated *at least* all changes up until serial X  (but maybe also partial changes from X+n).
> 
> I'm afraid that the mirrors data are now inconsistent - we can repair that once we have a stable mirroring approach again, but until then people will start getting annoyed again. 
> 
> I'm also concerned that I don't really have time to follow up on what's happening with TUF regarding mirroring on top of what happened regarding the CDN. My feeling is that will result in more fire fighting.
> 
> So - what's the next step that can happen ASAP?

The immediate way to get around the CDN/mirroring problems and to revert
to the pre-CDN consistency level, is to use the same access that fastly 
uses to get updates from pypi.python.org, namely a request on front.python.org
with a host-header.  I have this info from Donald with the cave-eat that
it's not guaranteed to remain possible.  Maybe Noah could agree to not
remove this facility without the current actors being on board for changes?
(i am also fine to have a dedicated domain instead of course).

Once this is settled, we can move on to fix current tools and deployments
and afterwards think about future improvements without the current urgency.

holger


> Christian
> 
> -- 
> Christian Theune · ct at gocept.com
> gocept gmbh & co. kg · Forsterstraße 29 · 06112 Halle (Saale) · Germany
> http://gocept.com · Tel +49 345 1229889-7
> Python, Pyramid, Plone, Zope · consulting, development, hosting, operations
>