urllib2 redirect error

[Steven D'Aprano]

On Tue, 02 Sep 2014 02:08:47 -0400, Sumit Ray wrote:

> Hi,
> 
>  I've tried versions of the following but continue to get errors:
> 
> ------------- snip -------------
> url = 'https://www.usps.com/send/official-abbreviations.htm' 
> request = urllib2.build_opener(urllib2.HTTPRedirectHandler).open(url)
> ------------- snip -------------
> 
> Generates an exception:
> urllib2.HTTPError: HTTP Error 301: The HTTP server returned a redirect
> error that would lead to an infinite loop. The last 30x error message
> was:
> Moved Permanently

I'm not an expert, but that sounds like a fault at the server end. I just
tried it in Chrome, and it worked, and then with wget, and I get the same
sort of error:


steve at runes:~$ wget https://www.usps.com/send/official-abbreviations.htm
--2014-09-02 17:22:31--  https://www.usps.com/send/official-abbreviations.htm
Resolving www.usps.com... 23.9.230.219, 2600:1415:8:181::1bf2, 2600:1415:8:183::1bf2
Connecting to www.usps.com|23.9.230.219|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.usps.com/root/global/server_responses/webtools-msg.htm [following]
--2014-09-02 17:22:31--  https://www.usps.com/root/global/
server_responses/webtools-msg.htm
Reusing existing connection to www.usps.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.usps.com/root/global/server_responses/webtools-msg.htm [following]
[...]
--2014-09-02 17:22:32--  https://www.usps.com/root/global/server_responses/webtools-msg.htm
Reusing existing connection to www.usps.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.usps.com/root/global/server_responses/webtools-msg.htm [following]
20 redirections exceeded.


Sounds like if the server doesn't recognise the browser, it gets 
confused and ends up in a redirect loop.

You could try setting the user-agent and see if that helps:


steve at runes:~$ wget -U "firefox 9999 (mozilla compatible)" https://www.usps.com/send/official-abbreviations.htm
--2014-09-02 17:28:03--  https://www.usps.com/send/official-abbreviations.htm
Resolving www.usps.com... 23.9.230.219, 2600:1415:8:183::1bf2, 2600:1415:8:181::1bf2
Connecting to www.usps.com|23.9.230.219|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `official-abbreviations.htm'

    [  <=>                                  ] 102,670      260K/s   in 0.4s    

2014-09-02 17:28:05 (260 KB/s) - `official-abbreviations.htm' saved [102670]




-- 
Steven