Fastest way to retrieve and write html contents to file

[DFS]

On 5/2/2016 12:49 AM, Ben Finney wrote:
> DFS <nospam at dfs.com> writes:
>
>> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for 10
>> iterations, vs 0.88 for python.
>>
>> […]
>>
>> urllib2 and requests were about the same speed as urllib.urlretrieve,
>> while pycurl was significantly slower (1.2 seconds).
>
> Network access is notoriously erratic in its timing. The program, and
> the machine on which it runs, is subject to a great many external
> effects once the request is sent — effects which will significantly
> alter the delay before a response is completed.
>
> How have you controlled for the wide variability in the duration, for
> even a given request by the *same code on the same machine*, at
> different points in time?
>
> One simple way to do that: Run the exact same test many times (say,
> 10 000 or so) on the same machine, and then compute the average of all
> the durations.
>
> Do the same for each different program, and then you may have more
> meaningfully comparable measurements.


I tried the 10-loop test several times with all versions.

The results were 100% consistent: VBSCript xmlHTTP was always 2x faster 
than any python method.