[Catalog-sig] Mirror list detection/construction - PEP 381

[P.J. Eby]

At 08:59 AM 7/22/2010 +0100, Martin v. Löwis wrote:
>>>Given the fragility of this it seems that we might want to consider
>>>alternative mirrorlist discovery mechanism.
>>
>>As a fallback, you can of course probe addresses on a binary search, or
>>even just select a random mirror in the first place. (If you overshoot
>>the list, you just decrease the max on your binary lookup.)
>
>I think you misunderstood the question. The issue is not at all what 
>mirrors to use once you know what they are - the question is how to find
>out what mirrors exist in the first place.

Yes -- which can be done by probing.  If, say, 'x.pypi.python.org' 
doesn't exist, that gives you an upper bound on how many mirrors must exist.

If you are only going to choose a random mirror anyway, then simply 
choosing one at random to start (and adjusting your limit down if you 
overshoot) will accomplish both at the same time.


>When you know what they are, I do support the idea of trying them all
>simultaneously, though not in parallel. With sockets, you can have many
>open without having to use threads.

Interesting.  I wonder how easy that will be to do in a 
cross-platform manner; I seem to recall that Twisted had some rather 
convoluted code to set socket options correctly for this sort of 
thing on different platforms.

Of course, if you are doing this, then you could also simply begin at 
'a' and add hosts to your list until you either get a failed lookup, 
or you get an IP that matches that of 'last.pypi.python.org'.


>I'll post some code that does that shortly.
>
>Regards,
>Martin