[Catalog-sig] High-availability for PyPI, mirroring infrastructures?

[Tarek Ziade]

2008/8/12 Phillip J. Eby <pje at telecommunity.com>

> At 02:13 PM 8/12/2008 +0200, Tarek Ziade wrote:
>
>> 2008/8/12 Jim Fulton <<mailto:jim at zope.com>jim at zope.com>
>>
>> On Aug 12, 2008, at 4:41 AM, Tarek Ziade wrote:
>> ...
>>
>> Since mirrors for PyPI and alternative egg servers are being , what about
>> making the "index" variable in
>> zc.buildout accept multiple values, like what find-links does ?
>>
>>
>>
>> Buildout uses and tries hard to be compatible with setuptools. In
>> particular, it relies on the setuptools.package_index.PackageIndex class,
>> which only allows a single index to be provided.
>>
>>
>> Yes, I was thinking of handling several PackageIndex instances on
>> zc.buildout side as a first thaught.
>>
>
> What you really want/need is to make the PackageIndex support retrieval
> from multiple index urls; the PackageIndex itself aggregates available
> packages from sources such as the local file system, -f urls, and an
> underlying package index.  So having multiple aggregators would duplicate
> processing, and deprive you of a global ordering of package precedences.
>
>
Is this a feature you would like to see in setuptools ? If so I can write a
patch,

Besides this feature, there's one feature we started to add on zc.buildout
but could be put in setuptools's PackageIndex as well I believe :

Adding timeouts for url openings, to speed up the processing. For instance,
when several urls are visited for
a given package, and when a server pointed by one of the url is down, it can
last forever.

urlib2.urlopen does not have a timeout option yet, even if it has been
discussed for Python 2.6 inclusion, 2 years ago  [1]
So we used *socket*.*setdefaulttimeout*() , which looks safe to change in
this use case.

At zc.buildout level, we could wait for up to 5 minutes to finish a buildout
upgrade,
and setting the timeout to a few seconds solved the problem efficiently.

In a shape of a decorator, that could be :

# five seconds
TIMEOUT = 5

def timedout(func):
    def _timedout(*args, **kw):
        old =socket.getdefaulttimeout()
        socket.setdefaulttimeout(TIMEOUT)
        try:
            return func(*args, **kw)
        finally:
            socket.setdefaulttimeout(old)
    return _timedout


Tarek

[1] http://mail.python.org/pipermail/python-dev/2006-July/066965.html

-- 
Tarek Ziadé - Directeur Technique
INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632
Bureaux de la Colline - 1 rue Royale - Bâtiment D - 9ème étage
92210 Saint Cloud - France
Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04
http://www.ingeniweb.com - une société du groupe Alter Way
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20080813/977c1329/attachment.htm>