[issue26416] Deprecate the regex_v8, telco, and spectral_norm benchmarks

[Antoine Pitrou]

Antoine Pitrou added the comment:

> On the speed mailing list, it was also suggested to use the geometric mean rather than the minimum or the average.

This should be considered a bit more carefully.

First, geometric mean is only useful when you are agregating heterogenous numbers. Here, we are agregating homogenous numbers (results from a single benchmark), so the arithmetic mean should be preferred.

Second, there still is the issue of weeding out outliars (due to e.g. background activity). So perhaps the 20% slowest runs should be discarded.

Third, to get enough precision in the arithmetic mean, the number of individual runs (separate process invocations, to smoothen variabilities due to cache aliasing etc.) should be raised to a sufficient number. See the central limit theorem.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26416>
_______________________________________