[Catalog-sig] PyPI down again...

[M.-A. Lemburg]

Tarek Ziadé wrote:
> On Fri, Jun 11, 2010 at 11:06 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> "Martin v. Löwis" wrote:
>>> For a smaller project, start putting mirror support into setuptools or
>>> distribute; this would make short (several hours) outages less severe
>>> for the class of users that want permanent availability for downloading.
>>> It's unlikely that the mirrors would break when the master goes down;
>>> they just stop mirroring.
>>
>> A better and cleaner strategy is to put the static PyPI information
>> up on Amazon Cloudscape and have DNS take care of providing local
>> mirrors (edge servers) to setuptools et al.
>>
>> Such a setup won't require any complicated mirror logic in any
>> of the existing client tools.
>>
>> By moving the PyPI installation to Amazon AWS, we could also
>> get the RPC access distributed to more than just one server.
>>
>> As I said before, the PSF infrastructure committee needs to get on
>> of the job of getting this implemented (including funding this
>> development).
>>
>> If someone wants to volunteer helping with the setup, please contact
>> the PSF at psf at python.org.
> 
> What about continuing the work that was started last year ?
> (and not finished due to a lack of time)
> 
> There's a PEP we have started about a mirroring infrastructure:
> http://www.python.org/dev/peps/pep-0381/
> 
> Some of its parts are already implemented in PyPI, and
> what we need now is to work on the client side (pip, distribute, etc)
> and bootstrap one or two mirrors using the protocol.

We've had some private discussions about this, so I'm just
going to summarize...

The idea here is not to override the mirror PEP ideas,
but to use the existing PyPI installation and put the
content needed for the most widely distributed package tool
(currently setuptools and zc.buildout) on a content
delivery network (CDN) in order to have it highly available
on a managed edge network.

Amazon Cloudfront is such a CDN and has Python interfaces,
hence the idea to use Cloudfront.

I asked for volunteers, because I didn't know enough about
Amazon Cloudfront to write up a proposal and don't have
the cycles available to implement such a setup myself.

In the meantime, I've done some research and now know
enough to write a proposal for the PSF board to consider.
If the board thinks it's a good idea, we'll need to
pursue finding volunteers to implement it.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 13 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK                35 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/