[issue45261] Unreliable (?) results from timeit (cache issue?)

[Marc-Andre Lemburg]

Marc-Andre Lemburg <mal at egenix.com> added the comment:

On the topic average vs. minimum, it's interesting to see this pop up
every now and then. When I originally wrote pybench late in 1997, I used
average, since it gave good results on my PC at the time.

Later on, before pybench was added to Tools/ in Python 2.5 in 2006, people
started complaining about sometimes getting weird results (e.g. negative
times due to the calibration runs not being stable enough). A lot of noise
was made from the minimum fans, so I switched to minimum, which then made
the results more stable, but I left in the average figures as well.

Then some years later, people complained about pybench not being
good enough for comparing to e.g. PyPy, since those other implementations
optimized away some of the micro-benchmarks which were used in pybench.
It was then eventually removed, to not confuse people not willing
to try to understand what the benchmark suite was meant for, nor
understand the issues around running such benchmarks on more modern
CPUs.

CPUs have advanced a lot since the days pybench was written and so
reliable timings are not easy to get unless you invest in dedicated
hardware, custom OS and CPU settings and lots of time to calibrate
everything. See Victor's research for more details.

What we have here is essentially the same issue. timeit() is mostly
being used for micro-benchmarks, but those need to be run in dedicated
environments. timeit() is good for quick checks, but not really up
to the task of providing reliable timing results.

One of these days, we should ask the PSF or one of its sponsors to
provide funding and devtime to set up such a reliable testing
environment. One which runs not only high end machines, but also
average and lower end machines, and using different OSes,
so that we can detect performance regressions early and easily
on different platforms.

----------
nosy: +lemburg

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45261>
_______________________________________