urllib2 rate limiting

[Dimitrios Apostolou]

On Thursday 10 January 2008 22:42:44 Rob Wolfe wrote:
> Dimitrios Apostolou <jimis at gmx.net> writes:
> > On Thu, 10 Jan 2008, Rob Wolfe wrote:
> >> Dimitrios Apostolou <jimis at gmx.net> writes:
> >>> P.S. And something simpler: How can I disallow urllib2 to follow
> >>> redirections to foreign hosts?
> >>
> >> You need to subclass `urllib2.HTTPRedirectHandler`, override
> >> `http_error_301` and `http_error_302` methods and throw
> >> `urllib2.HTTPError` exception.
> >
> > Thanks! I think for my case it's better to override redirect_request
> > method, and return a Request only in case the redirection goes to the
> > same site. Just another question, because I can't find in the docs the
> > meaning of (req, fp, code, msg, hdrs) parameters. To read the URL I
> > get redirected to (the 'Location:' HTTP header?), should I check the
> > hdrs parameter or there is a better way?
>
> Well, according to the documentation there is no better way.
> But I looked into the source code of `urllib2` and it seems
> that `redirect_request` method takes one more parameter
> `newurl`, what is probably what you're looking for. ;)
>
> Regards,
> Rob

Cool! :-) Sometimes undocumented features provide superb solutions... I wonder 
if there is something similar for rate limiting :-s


Thank you,
Dimitris