[Distutils] Sourceforge changed HTML again...

[Phillip J. Eby]

At 08:27 AM 1/23/2006 -0500, Charlie Moad wrote:
>On 1/18/06, Ian Bicking <ianb at colorstudy.com> wrote:
> > Phillip J. Eby wrote:
> > > I checked what the NetBSD "pkgsrc" system does, and it uses the fact
> > > that there is a .dl.sourceforge.net subdomain for mirrors.  I
> > > investigated further and found that dl.sourceforge.net is a round-robin
> > > (or random?) DNS for each of the mirrors in the subdomain, so there's a
> > > simple transformation from the user-visible download pages to the actual
> > > download address.  You can see this in the new fix_sf_url() function I
> > > added to setuptools.package_index.
> >
> > I just got a Connection Refused error, but it worked on the second try.
> >   As I remember, the port system typically has a set of links, and
> > frequently fails over from one link to the next, so I expect that this
> > error should be expected.  When it is encountered, setuptools should
> > just try again until it works.  I have noticed in the past that SF
> > mirrors go up and down quite frequently, probably too fast for DNS to
> > keep up.
>
>Does the pypi download url need to be updated to be compatible with
>svn setuptools?  I am assuming no, but I am constantly getting 404's
>from easy_install, yet I can paste the dl.sf.net link into a browser
>and it works?!  These files have been on sf long enough to be
>propgated around, and constantly ~= 20 attempts on the command line.
>I am seeing this with matplotlib, btw.

Interesting.  I just tried "easy_install matplotlib" and it worked the 
first time.  I actually have never yet gotten a 404 or other problem using 
the dl.sf.net URLs on any package.

However, a little experimentation shows that my Windows PC caches the DNS 
reply for an extended period, so I guess that means that if you get a bad 
mirror IP, you're stuck with it until the cache clears or you run "ipconfig 
/flushdns".  (In contrast, my Linux machine returns a different IP every 
time the name is looked up.)

A little experimentation with the socket module shows that I can get the 
full list of mirror IPs from Python, so I've changed setuptools in SVN to 
just randomly select one to use, which should fix the sticking problem on 
Windows (and any other platform where it occurs).

It's not a perfect solution, since of course you can still end up with a 
bad mirror for some download, but I'm not sure what else to do, short of 
having some kind of option/configuration to control mirror selection.