httplib incredibly slow :-(

Chris Withers chris at simplistix.co.uk
Tue Aug 11 16:25:47 EDT 2009


Hi All,

I'm using the following script to download a 150Mb file:

from base64 import encodestring
from httplib import HTTPConnection
from datetime import datetime

conn = HTTPSConnection('localhost')
headers = {}
auth = 'Basic '+encodestring('username:password').strip()
headers['Authorization']=auth
t = datetime.now()
print t
conn.request('GET','/somefile.zip',None,headers)
print 'request:',datetime.now()-t
response = conn.getresponse()
print 'response:',datetime.now()-t
data = response.read()
print 'read:',datetime.now()-t

The output shows it takes over 20 minutes to do this.
However, this is on a local network, and downloading the same file in IE 
takes under 3 seconds!

I saw this issue:

http://bugs.python.org/issue2576

I tried changing the buffer size to 4096 in a subclass as the issue 
suggested, but I didn't see the reported speed improvement.
I'm using Python 2.6.2.

Does anyone know of an alternative library for creating http requests 
and getting their responses that's faster but hopefully has a similar 
interface?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk



More information about the Python-list mailing list