urllib2 performance on windows, usb connection

[dq]

MRAB wrote:
> dq wrote:
>> MRAB wrote:
>>> dq wrote:
>>>> Martin v. Löwis wrote:
>>>>>> So does anyone know what the deal is with this?  Why is the
>>>>>> same code so much slower on Windows?  Hope someone can tell
>>>>>> me before a holy war erupts :-)
>>>>>
>>>>> Only the holy war can give an answer here. It certainly has 
>>>>> *nothing* to do with Python; Python calls the operating system
>>>>> functions to read from the network and write to the disk almost
>>>>> directly. So it must be the operating system itself that slows
>>>>> it down.
>>>>>
>>>>> To investigate further, you might drop the write operating, and
>>>>> measure only source.read(). If that is slower, then, for some
>>>>> reason, the network speed is bad on Windows. Maybe you have the
>>>>> network interfaces misconfigured? Maybe you are using wireless
>>>>> on Windows, but cable on Linux? Maybe you have some network
>>>>> filtering software running on Windows? Maybe it's just that
>>>>> Windows sucks?-)
>>>>>
>>>>> If the network read speed is fine, but writing slows down, I
>>>>> ask the same questions. Perhaps you have some virus scanner
>>>>> installed that filters all write operations? Maybe Windows
>>>>> sucks?
>>>>>
>>>> Thanks for the ideas, Martin.  I ran a couple of experiments to find 
>>>> the culprit, by downloading the same 20 MB file from the
>>>> same fast server. I compared:
>>>>
>>>> 1.  DL to HD vs USB iPod.
>>>> 2.  AV on-access protection on vs. off
>>>> 3.  "source. read()" only vs.  "file.write( source.read() )"
>>>>
>>>> The culprit is definitely the write speed on the iPod.  That is, 
>>>> everything runs plenty fast (~1 MB/s down) as long as I'm not
>>>> writing directly to the iPod.  This is kind of odd, because if I
>>>> copy the file over from the HD to the iPod using windows
>>>> (drag-n-drop), it takes about a second or two, so about 10 MB/s.
>>>>
>>>> So the problem is definitely partially Windows, but it also seems
>>>> that Python's file.write() function is not without blame.  It's
>>>> the combination of Windows, iPod and Python's data stream that is
>>>> slowing me down.
>>>>
>>>> I'm not really sure what I can do about this.  I'll experiment a
>>>> little more and see if there's any way around this bottleneck.
>>>> If anyone has run into a problem like this, I'd love to hear
>>>> about it...
>>>>
>>> You could try copying the file to the iPod using the command line,
>>> or copying data from disk to iPod in, say, C, anything but Python.
>>> This would allow you to identify whether Python itself has anything
>>> to do with it.
>>
>> Well, I think I've partially identified the problem.  target.write( 
>> source.read() ) runs perfectly fast, copies 20 megs in about a
>> second, from HD to iPod.  However, if I run the same code in a while
>> loop, using a certain block size, say target.write( source.read(4096)
>> ), it takes forever (or at least I'm still timing it while I write
>> this post).
>>
>> The mismatch seems to be between urllib2's block size and the write 
>> speed of the iPod, I might try to tweak this a little in the code and
>> see if it has any effect.
>>
>> Oh, there we go:   20 megs in 135.8 seconds.  Yeah... I might want to
>> try to improve that...
>>
> How long does it take to transfer 4KB? If it can transfer 1MB/s then I'd
> say that 4KB is too small. Generally speaking, the higher the data rate,
> the larger the blocks you should be transferring at a time, IMHO.
> 
> You could write a script to test the transfer speed with different block
> sizes.

Thanks MRAB, 32 or 64 KB seems to be quickest, but I'll do a more 
scientific test soon and see what turns up.