[Distutils] What to do about the PyPI mirrors

[Noah Kantrowitz]

On Aug 5, 2013, at 11:09 PM, Christian Theune <ct at gocept.com> wrote:

> Hi,
> 
> looks like I'm late to the party to figure out that I'm going to be hurt again.
> 
> I'd like to suggest explicitly considering what is going to break due to this and how much work you are forcefully inflicting on others. My whole experience around the packaging (distribute/setuptools) and mirroring/CDN in this year estimates cost for my company somewhere between 10k-20k EUR just for keeping up with the breakage those changes incure. It might be that we're wonderfully stupid (..enough to contribute) and all of this causes no headaches for anybody else …. Overall, guessing that the packaging infrastructure is used by probably multiple thousands of companies then I'd expect that at least 100 of them might be experiencing problems like us. Juggling arbritrary numbers I can see that we're inflicting around a million EURs of cost that nobody asked for. 
> 
> More specific statements below.
> 
> On 2013-08-04 22:25:01 +0000, Donald Stufft said:
> 
> Here's my PEP for Deprecating and  Removing the Official Public Mirrors
> 
> It's source is at: https://github.com/dstufft/peps/blob/master/mirror-removal.rst
> 
> Abstract
> =======
> This PEP provides a path to deprecate and ultimately remove the official
> public mirroring infrastructure for `PyPI`_. It does not propose the removal
> of mirroring support in general.
> 
> -1 - maybe I don't have the right to speak up on CDN usage, but personally I feel it's a bad idea to delegate overall PyPI availability exclusively to a commercial third party. It's OK for me that we're using them to improve PyPI availability, but completely putting our faith in their hands, doesn't sound right to me.
> 
> Rationale
> ========
> The PyPI mirroring infrastructure (defined in `PEP381`_) provides a means to
> mirror the content of PyPI used by the automatic installers. It also provides
> a method for autodiscovery of mirrors and a consistent naming scheme.
> 
> There are a number of problems with the official public mirrors:
> 
> * They give control over a \*.python.org domain name to a third party,
>   allowing that third party to set or read cookies on the pypi.python.org and
>   python.org domain name.
> 
> Agreed, that's a problem.
> 
> * The use of a sub domain of pypi.python.org means that the mirror operators
>   will never be able to get a certificate of their own, and giving them
>   one for a python.org domain name is unlikely to happen.
> 
> Agreed.
> 
> * They are often out of date, most often by several hours to a few days, but
>   regularly several days and even months.
> 
> That's something that the mirroring infrastructure should have been constructed for. I completely agree that the way the mirroring was established was way sub-optimal. I think we can do better.
> 
> * With the introduction of the CDN on PyPI the public mirroring infrastructure
>   is not as important as it once was as the CDN is also a globally distributed
>   network of servers which will function even if PyPI is down.
> 
> Well, now we have one breakage point more which keeps annoying me. This argument is not completely true. They may be getting better over time but we have invested heavily to accomodate the breakage - that needs to be balanced with some benefit in the near future.

To be clear, the CDN and other server-side improvements are not a hard-HA replacement like a local company mirror. You are exactly the use case that can and should be using a mirror for your own use. We are doing _nothing_ that disrupts this use case and will support is exactly as before.

> 
> * Although there is provisions in place for it, there is currently no known
>   installer which uses the authenticity checks discussed in `PEP381`_ which
>   means that any download from a mirror is subject to attack by a malicious
>   mirror operator, but further more due to the lack of TLS it also means that
>   any download from a mirror is also subject to a MITM attack.
> 
> Again, I think that was a mistake during the introduction of the mirroring infrastructure: too few people, too confusing PEP. 
> 
> * They have only ever been implemented by one installer (pip), and its
>   implementation, besides being insecure, has serious issues with performance
>   and is slated for removal with it's next release (1.5).
> 
> Only if you consider the mirror auto-discovery protocol. I'm not sure whether using DNS was such a smart move. A simple HTTP request to find mirrors would have been nice. I think we can still do that.
> 
> Also, not everyone wants or needs auto-detection the way that the protocol describes it. I personally just hand-pick a mirror (my own, hah) and keep using that. 
> 
> We are also thinking about providing system-level default configuration to hint tools like PIP and setuptools to a different default index that is closer from a network perspective. From a customer perspective this should be "PyPI".
> 
> I'd like to avoid breakage. Again, if you don't let me choose where to spend my time, I'd rather invest the time I need for cleaning up the breakage into something constructive.
> 
> The indices are in active use. f.pypi.python.org is seeing between 150-300GB of traffic per month, the patterns widely ranging over the last month. This is traffic that is not used internally from gocept.
> 
> Due to the number of issues, some of them very serious, and the CDN which more
> or less provides much of the same benefits this PEP proposes to first
> deprecate and then remove the public mirroring infrastructure. The ability to
> mirror and the method of mirroring will not be affected and the existing
> public mirrors are encouraged to acquire their own domains to host their
> mirrors on if they wish to continue hosting them.
> 
> The biggest benefit of the mirroring infrastructure is that it is intended to be de-centralized.
> As a community member I can step up and take over responsibility of availability, performance, and security of a mirror.
> 
> As a community member I have to completely submit to whatever the CDN does and contacting another community member who hopefully will be with us for a long time and stay in good contact with the CDN for us. That's centralization and I don't like that a bit.
> 
> Plan for Deprecation & Removal
> =============================
> Immediately upon acceptance of this PEP documentation on PyPI will be updated
> to reflect the deprecated nature of the official public mirrors and will
> direct users to external resources like http://www.pypi-mirrors.org/ to
> discover unofficial public mirrors if they wish to use one.
> 
> On October 1st, 2013, roughly 2 months from the date of this PEP, the DNS names
> of the public mirrors ([a-g].pypi.python.org) will be changed to point back to
> PyPI which will be modified to accept requests from those domains. At this
> point in time the public mirrors will be considered deprecated.
> 
> Then, roughly 2 months after the release of the first version of pip to have
> mirroring support removed (currently slated for pip 1.5) the DNS entries for
> [a-g].pypi.python.org and last.pypi.python.org will be removed and PyPI will
> no longer accept requests at those domains.
> 
> Oh great. That means in about 4 months I have to go through *any installation that my company maintains* and sift through whether we're still referencing f.pypi.python.org anywhere.
> 

Yes, sorry, no matter how we procedurally handle this shutdown, this will be required. From doing this myself more times than I care to remember, I've not found the actual time difference between ~one month and a year+ mattering much, but having to do it at all is unlikely to change, just when and where.

> Can I write a check? 
> 
> Unofficial Public or Private Mirrors
> ===================================
> The mirroring protocol will continue to exist as defined in `PEP381`_ and
> people are encouraged to utilize to host unofficial public and private mirrors
> if they so desire. For operators of unofficial public or private mirrors the
> recommended mirroring client is `Bandersnatch`_.
> 
> Thanks for the recommendation.
> 
> Instead of this dance breaking many things yet again, I'd love if we could find a way forward keeping the infrastructure.
> 
> Some ideas:
> 
> - Take control of *.pypi.python.org back
> - Record other public names of the mirrors
> - Use 301 redirects to send old installations over to the new mirror names.
> - Make it easier for community members to help maintain the list of mirrors.
> - Make a better (faster) removal policy of mirrors if the owners are not responsive.
> - Make it easier for other community members to set up and maintain mirrors. I'm happy to improve bandersnatch where needed.


Between now and the first DNS change, I would absolutely recommend any current public mirrors to redirect users to their new domain name if they intend to have one, and we'll do whatever we can to help make users aware of the switch. I would rather have a clear timeline with fewer steps than add another stage where we (PSF) are issuing redirects to non-PSF servers. Very very +1 on the easier bandersnatch-ing though, I really would love to see more mirrors out there, I just don't want them associated with PyPI or python.org, and I don't want pip to be trying to auto-discover them. I am also hoping that pypi-mirrors.org will continue to operate as a community project (side note, I would be happy to assist with hosting for it if Ken reads this list and if thats a concern of his) and that the mirror operators can develop policies for things like this. I defer to Nick and Ken if they would like distutils-sig to be involved in that process, but as it stands Ken can apply whatever rules he wants to his mirror list.

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130805/4b1a7690/attachment.pgp>