Progress Bar with urllib2

Andrew Godwin andrewgodwin at blueyonder.co.uk
Tue Apr 26 16:28:43 EDT 2005


I'm trying to write a python script to download data (well, files) from a HTTP server (well, a PHP script spitting them out, at least).
The file data is just the returned data from the request (the server script echoes the file and then dies).

I call the page using urllib2, like so:

satelliteRequest = urllib2.Request(satelliteServer + "?command=download&filepath="+filepath)
satelliteRequestData = {"username":satelliteUsername, "password":satellitePassword}
satelliteRequest.add_data(urllib.urlencode(satelliteRequestData))
satelliteOpener = urllib2.build_opener()
satelliteOpener.addheaders = [('User-agent', userAgent)]

Now, if I want to download the file all at once, I just do

satelliteData = satelliteOpener.open(satelliteRequest).read()

But some of these files are going to be really, really big, and I want to get a progress bar going. 
I've tried doing a while loop like this:

chunkSize = 10240
while 1:
 dataBuffer = satelliteOpener.open(satelliteRequest).read(chunkSize)
 data += dataBuffer
 if not dataBuffer:
  break

But that just gives me the first 10240 bytes again and again. Is there something I'm missing here?
It might even be I'm calling urllib2 the wrong way (does it download when you read() or when you create the Request?)

All help is appreciated, I'm sort of stuck here.

Andrew Godwin



More information about the Python-list mailing list