[Catalog-sig] PyPI down again...

[M.-A. Lemburg]

"Martin v. Löwis" wrote:
>> I think it overlaps a bit the PEP goal, which is to set up a network
>> of mirrors,
>> and have them listed in the PyPI DNS  so clients can switch from one
>> mirror
>> to another.(and even do geoloc!)
> 
> JFTR, this already exists. a.mirrors.pypi.python.org and
> b.mirrors.pypi.python.org are already there and could be used by clients.
> 
>> Well maybe this is the best path to follow right now, as it will be
>> done faster,
>> without having to interact with much people to set it up, so it's a
>> quick win.
> 
> My main worry (besides the client integration) is statistics: I do want
> to get download statistics. So anybody implementing it would have to
> find a way of fetching the download numbers from Amazon.

Download statistics are readily available from Amazon Cloudfront,
so no worries: you'll get statistics for all edge server downloads.

http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/index.html?AccessLogs.html

>> But it will probably kill the mirroring protocol idea from the PEP in
>> the process,
>> which I think is superior in the long term since it provides a
>> standardized ground
>> for the community to set up mirrors independently from pypi.python.org.
> 
> I also remain skeptical that this cloud idea is useful at all. Amazon
> Cloudfront is a *beta* service. So they aren't sure themselves whether
> it works correctly - and there have been reports about two-day outages
> of EC2, for bitbucket.org. There also have been complaints about the
> available bandwidth. So I'm not sure whether replacing a single point of
> failure with a different one is actually improving anything.

Amazon Cloudfront uses S3 as basis for the service, S3 has been
around for years and has a very stable uptime:

http://www.readwriteweb.com/archives/amazon_s3_exceeds_9999_percent_uptime.php

Cloudfront itself has been around since Nov 2008.

You can check their current online status using this panel:

http://status.aws.amazon.com/

Apart from the gained availability and outsourced management,
we'd also get faster downloads in most parts of the world,
due to the local caching Cloudfront is applying (and this can
be used to further increase the availability, since we can
control the expiry time of those local copies).

So in summary we are replacing a single point of failure with N
points of failure (with N being the number of edge caching
servers they use).

Regaring the bitbucket problem you mentioned:

EC2 is their virtual server service, which we don't use. The
bitbucket problems originated from a) a DDoS attack on their
virtual servers running on EC2 and b) a problem with the
Amazon EBS, which is their virtualized SAN, and was related
to the way the DDoS was done (EBS and the DDoS attack both
used UDP):

http://blog.bitbucket.org/2009/10/04/on-our-extended-downtime-amazon-and-whats-coming/
"""
And to re-iterate, the problem wasn’t really Amazon EC2 or EBS, it was isolated to our case, due to
the nature of the attack.
"""

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 14 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK                34 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/