Fastest way to write huge files

[Larry Bates]

Mohamed Yousef wrote:
>> If connection is over Internet via HTTP the connection speed is so slow in
>> relation to the speed of your CPU that it doesn't really matter.
> this is not always true , espicially when using a localhost or a local
> netwtork server
> the problem is the increase in cpu and memory usage make it a painful bug
> such that downloading a big file (say 1 GB ) would introduce a big cpu
> usage (already tested)
> 
> Thanks ,
> 
> Regards,
> Mohamed Yousef

I disagree.  There is very little overhead in file writing if you stream your 
writes to disk in small blocks as they arrive via HTTP.  Don't wait for the 
entire 1Gb to arrive and then write it.  Python can write small to a file 
blazingly fast with normal file writing I/O. Any extra CPU overhead you may see 
is probably due to reading the entire 1Gb into memory and seeing swapping to 
disk as you exhaust main memory.  Interleaving your HTTP reading with file 
writing should be very fast.

-Larry