[Distutils] PyPI CDN Updates For Greater Availability

[Donald Stufft]

Several changes were just deployed to PyPI's CDN. The general
theme behind the changes is making it so that PyPI appears as
functional as possible through a failure of the server hosting it. This
should increase the availability of PyPI and enable things such as
installation and browsing the site to continue to work through a 
catastrophic host failure on the PSF infrastructure.

The details of what changes are:

- Anonymous users will find that /pypi* pages are now cached for
   a short amount of time (currently 60s).

- Objects will be stored in the cache for some time past their
  expiration  date. They will not be used except in two circumstances:
  - A request is taking longer than 15s to complete, a "stale" object
    will be returned to prevent a pile up from occurring.
  - The backend[1] has been deemed unhealthy, in which case stale
     objects will be served in order to allow some level of functionality
     until the backend has been restored.

- In the advent of an unhealthy backend all requests will be forced to
   be anonymous, making them eligible for the stale objects that
   have been cached.

- The /mirrors and /security pages will be cached for a week, allowing
   them to likely be available through a backend failure making it easy
   to locate mirrors[2] or report a security issue.

- Miscellaneous changes to normalize various things so that a single
  item in the cache will be able to be used for more requests, making
  it more likely that any particular request will be served from the Cache.


[1] Backend in this context means the server hosting PyPI itself, what
     the CDN itself connects too.
[2] Using the mirrors is done so at your own risk. None of the tools
     currently verify the downloads and they are downloaded over
     HTTP. This makes it trivial for an attacker to execute arbitrary
     code on your machine via a MITM.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130704/7fe7c544/attachment.pgp>