urllib2 performance on windows, usb connection

[MRAB]

dq wrote:
> MRAB wrote:
>> dq wrote:
>>> Martin v. Löwis wrote:
>>>>> So does anyone know what the deal is with this?  Why is the
>>>>> same code so much slower on Windows?  Hope someone can tell
>>>>> me before a holy war erupts :-)
>>>> 
>>>> Only the holy war can give an answer here. It certainly has 
>>>> *nothing* to do with Python; Python calls the operating system
>>>> functions to read from the network and write to the disk almost
>>>> directly. So it must be the operating system itself that slows
>>>> it down.
>>>> 
>>>> To investigate further, you might drop the write operating, and
>>>> measure only source.read(). If that is slower, then, for some
>>>> reason, the network speed is bad on Windows. Maybe you have the
>>>> network interfaces misconfigured? Maybe you are using wireless
>>>> on Windows, but cable on Linux? Maybe you have some network
>>>> filtering software running on Windows? Maybe it's just that
>>>> Windows sucks?-)
>>>> 
>>>> If the network read speed is fine, but writing slows down, I
>>>> ask the same questions. Perhaps you have some virus scanner
>>>> installed that filters all write operations? Maybe Windows
>>>> sucks?
>>>> 
>>> Thanks for the ideas, Martin.  I ran a couple of experiments to 
>>> find the culprit, by downloading the same 20 MB file from the
>>> same fast server. I compared:
>>> 
>>> 1.  DL to HD vs USB iPod.
>>> 2.  AV on-access protection on vs. off
>>> 3.  "source. read()" only vs.  "file.write( source.read() )"
>>> 
>>> The culprit is definitely the write speed on the iPod.  That is, 
>>> everything runs plenty fast (~1 MB/s down) as long as I'm not
>>> writing directly to the iPod.  This is kind of odd, because if I
>>> copy the file over from the HD to the iPod using windows
>>> (drag-n-drop), it takes about a second or two, so about 10 MB/s.
>>> 
>>> So the problem is definitely partially Windows, but it also seems
>>> that Python's file.write() function is not without blame.  It's
>>> the combination of Windows, iPod and Python's data stream that is
>>> slowing me down.
>>> 
>>> I'm not really sure what I can do about this.  I'll experiment a
>>> little more and see if there's any way around this bottleneck.
>>> If anyone has run into a problem like this, I'd love to hear
>>> about it...
>>> 
>> You could try copying the file to the iPod using the command line,
>> or copying data from disk to iPod in, say, C, anything but Python.
>> This would allow you to identify whether Python itself has anything
>> to do with it.
> 
> Well, I think I've partially identified the problem.  target.write( 
> source.read() ) runs perfectly fast, copies 20 megs in about a
> second, from HD to iPod.  However, if I run the same code in a while
> loop, using a certain block size, say target.write( source.read(4096)
> ), it takes forever (or at least I'm still timing it while I write
> this post).
> 
> The mismatch seems to be between urllib2's block size and the write 
> speed of the iPod, I might try to tweak this a little in the code and
> see if it has any effect.
> 
> Oh, there we go:   20 megs in 135.8 seconds.  Yeah... I might want to
> try to improve that...
> 
How long does it take to transfer 4KB? If it can transfer 1MB/s then I'd
say that 4KB is too small. Generally speaking, the higher the data rate,
the larger the blocks you should be transferring at a time, IMHO.

You could write a script to test the transfer speed with different block
sizes.