Strange performance issue

[Dan Stromberg]

Ulrich Eckhardt wrote:
> Dan Stromberg wrote:
>   
>> My new version formats an SD card and preallocates some file space in
>> about 3 minutes with "Optimize Performance" selected, and in about 30
>> minutes with "Optimize for Quick Removal" selected.  Needless to say, I
>> don't like the 27 minute penalty much.
>>     
>
> For performance, the driver will probably use delayed write operations,
> buffering etc. For quick removal, it will probably make all operations
> atomic, i.e. perform a write operation and not return until the data has
> really reached the SD card.
>   
The driver may delay writes, but I'd think it more likely that the 
filesystem or buffer cache would be doing so.
>> But the old version of the program formats and preallocates in 3 minutes
>> irrespective of whether the drive is in "optimize performance" or
>> "optimize for quick removal".
>>     
>
> Somehow, I guess the new version does many small writes while the old one
> doesn't. When optimizing for quick removal, those operations add up,
> otherwise their overhead is negligible.
>   
Nope, same block size.
>> one_meg = 'a'*2**20
>>     
>
> That's one mib, not one meg. ;)
>
>   
You're aware that a lot of people are ignoring the new units?
>> file = open('remove-me', 'w')
>>     
>
> Try adding the 'b' flag, too. I wouldn't expect it to affect the IO speed,
> but it is one factor that adds up.
This appears to have wholly accounted for the problem!  Thank you very much.

Adding a "b" to my open sped up the writes by a factor of about 15.
>  Otherwise, I looked over your program
> and couldn't find anything that would explain a problem. Just wondering,
> what transfer speed do you get with the two versions? What is the
> theoretical maximum for the SD card?
>   
I don't know what the theoretical max write speeds of the USB bus, card 
reader and card are, but I was getting over 10x faster using the same 
card and card reader on an ancient Thinkpad running Ubuntu.  With the 
"b" specified, the XP system is now faster than the Thinkpad.

Interesting that the "b" would make such a performance difference.  
You'd think it'd just be, in C, looking for newlines and adding a 
carriage return after them into a buffer of potentially double the size, 
then writing as usual.  The data I've been writing so far -has- no newlines.

Thanks!