urllib: an alternative URLOpener

[Mark Nottingham]

FWIW, I've put a fairly complete spidering class together as part of a
project... may be useful for somebody out there...


http://www.mnot.net/cacheability/download.html


----- Original Message -----
From: Jeremy Hylton <jeremy at cnri.reston.va.us>
Newsgroups: comp.lang.python
To: <python-list at cwi.nl>
Sent: Sunday, June 13, 1999 3:24
Subject: Re: urllib: an alternative URLOpener


> I've been working on-and-off on a replacement for urllib that is more
> easily customizable; they basic philosophy has been to avoid
> inheritance as the customization mechanism.  Instead, a single
> director class coordinates the activities of many small handlers that
> implement one feature/protocol.
>
> The resulting architecture makes it much easier to create a UrlOpener
> that handles redirects (which is probably the crucial feature of
> FancyUrlOpener) without getting all of the other cruft (like prompting
> the user to enter a password on stdin).
>
> I've been delayed a bit because I wanted to add better support for
> proxies, e.g. using a proxy only for URLs that match some pattern.
> Since people are asking about this, I'll expedite my release plans
> :-).  I'll try to make a useful release available early next week.
>
> Jeremy
>
>
>
>
>