Python urllib.urlretrieve vs. Perl getstore

[Adam S]

I just had a peek inside urllib.py and the urllib.urlretrieve
is essentially a FancyURLopener-retrieve.

It reads a 8*1024 block then writes it into the
file opened, see below:

fp is the opened url
tfp is the destination file.

    bs = 8*1024

    while block:
        tfp.write(block)
        block = fp.read(bs)

So really no more than  8*1024 bytes should never go into memory.
I've tried implementing my own version of this experimenting with
different block sizes but it makes little difference.

Adam

Dave Brueck wrote:
> 
> 40 MB files? Hmm... maybe is urllib pulling the entire file into memory?
> If so, the slowdown might have nothing to do with the network and
> everything to do with the OS swapping memory to disk.
> 
> -Dave