urllib2 rate limiting

[Rob Wolfe]

Dimitrios Apostolou <jimis at gmx.net> writes:

> On Thu, 10 Jan 2008, Rob Wolfe wrote:
>
>> Dimitrios Apostolou <jimis at gmx.net> writes:
>>
>>> P.S. And something simpler: How can I disallow urllib2 to follow
>>> redirections to foreign hosts?
>>
>> You need to subclass `urllib2.HTTPRedirectHandler`, override
>> `http_error_301` and `http_error_302` methods and throw
>> `urllib2.HTTPError` exception.
>
> Thanks! I think for my case it's better to override redirect_request
> method, and return a Request only in case the redirection goes to the
> same site. Just another question, because I can't find in the docs the
> meaning of (req, fp, code, msg, hdrs) parameters. To read the URL I
> get redirected to (the 'Location:' HTTP header?), should I check the
> hdrs parameter or there is a better way?

Well, according to the documentation there is no better way.
But I looked into the source code of `urllib2` and it seems
that `redirect_request` method takes one more parameter
`newurl`, what is probably what you're looking for. ;)

Regards,
Rob