[issue23693] timeit accuracy could be better

[STINNER Victor]

STINNER Victor added the comment:

Not only I'm too lazy to compute manually the number of loops and repeat, but also I don't trust myself. It's even worse when someone publishs results of a micro-benchmark. I don't trust how the benchmark was calibrated. In my experience, micro-benchmark are polluted by noise in timings, so results are not reliable. 

benchmarks.py calibration is based on time, whereas timeit uses hardcoded constants (loops=1000000, repeat=3) which can be modified on the command line.

benchmarks.py has 3 main parameters:

- minimum duration of a single run (--min-time): 100 ms by default
- maximum total duration of the benchmark: benchmark.py does its best to respect this duration, but it can be longer: 1 second by default
- minimum repeat: 5 by default

The minimum duration is increased if the clock resolution is bad (1 ms or more). It's the case on Windows for time.clock() on Python 2 for example. Extract of benchmark.py:

    min_time = max(self.config.min_time, timer_precision * 100)

The estimation of the number of loops is not reliable, but it's written to be "fast". Since I run a micro-benchmark many times, I don't want to wait too long. It's not a power of 10, but an arbitrary integer number. Usually, when running benchmark.py multiple times, the number of loops is different each time. It's not really a big issue, but it probably makes results more difficult to compare.

My constrain is max_time. The tested function may not have a linear duration (time = time_one_iteration * loops).

https://bitbucket.org/haypo/misc/src/348bfd6108e9985b3c2298d2745eb5ddfe7042e6/python/benchmark.py?at=default#cl-416

Repeat a test at least 5 times is a compromise between the stability of the result and the total duration of the benchmark.

Feel free to reuse my code to enhance time.py.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23693>
_______________________________________