Progress Bar with urllib2

[John A Ferguson]

On Tue, 26 Apr 2005 20:28:43 GMT, Andrew Godwin wrote:

> I'm trying to write a python script to download data (well, files) from a HTTP server (well, a PHP script spitting them out, at least).
> The file data is just the returned data from the request (the server script echoes the file and then dies).
> 
> I call the page using urllib2, like so:
> 
> satelliteRequest = urllib2.Request(satelliteServer + "?command=download&filepath="+filepath)
> satelliteRequestData = {"username":satelliteUsername, "password":satellitePassword}
> satelliteRequest.add_data(urllib.urlencode(satelliteRequestData))
> satelliteOpener = urllib2.build_opener()
> satelliteOpener.addheaders = [('User-agent', userAgent)]
> 
> Now, if I want to download the file all at once, I just do
> 
> satelliteData = satelliteOpener.open(satelliteRequest).read()
> 
> But some of these files are going to be really, really big, and I want to get a progress bar going. 
> I've tried doing a while loop like this:
> 
> chunkSize = 10240
> while 1:
>  dataBuffer = satelliteOpener.open(satelliteRequest).read(chunkSize)
>  data += dataBuffer
>  if not dataBuffer:
>   break
> 
> But that just gives me the first 10240 bytes again and again. Is there something I'm missing here?
> It might even be I'm calling urllib2 the wrong way (does it download when you read() or when you create the Request?)
> 
> All help is appreciated, I'm sort of stuck here.
> 
> Andrew Godwin

Each time through the loop you re-open the url and thus start from the
beginning. You need to separate the opening from the reading.

HTH,
 John