urllib2 httplib.BadStatusLine exception while opening a page on an Oracle HTTP Server

[Steven D'Aprano]

On Mon, 19 Jan 2009 13:00:44 -0800, ak wrote:

> Hi everyone,
> 
> I have a problem with urllib2 on this particular url, hosted on an
> Oracle HTTP Server
> 
> http://www.orange.sk/eshop/sk/portal/catalog.html?
type=post&subtype=phone&null
> 
> which gets 302 redirected to
> https://www.orange.sk/eshop/sk/catalog/post/phones.html, after setting a
> cookie through the Set-Cookie header field in the 302 reply. This works
> fin with firefox.
> 
> However, with urllib2 and the following code snippet, it doesn't work


Looking at the BadStatusLine exception raised, the server response line 
is empty. Looking at the source for httpllib suggests to me that the 
server closed the connection early. Perhaps it doesn't like connections 
from urllib2? 

I ran a test pretending to be IE using this code:

cookiejar = cookielib.LWPCookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
url = 'http://www.orange.sk/eshop/sk/portal/catalog.html?' \
    'type=post&subtype=phone&null'
agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; " \
    "NeosBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
headers = {'User-Agent': agent}
req = urllib2.Request(url, data=None, headers=headers)
try:
    s=opener.open(req)
except httplib.BadStatusLine, e:
    print e, e.line
else:
    print "Success"



but it failed. So the problem is not as simple as changing the user-agent 
string.

Other than that, I'm stumped.



-- 
Steven