[Speed] Median +- MAD or Mean +- std dev?

[Antoine Pitrou]

On Wed, 15 Mar 2017 08:54:44 +0200
Serhiy Storchaka <storchaka at gmail.com>
wrote:
> 
> But absolute values are not important for the purposes of our 
> benchmarking. We need only know whether one build is faster or slower 
> than others.

Not really.  If you don't know by how much it is faster or slower, it
is often useless in itself (because being 0.1% faster doesn't matter,
even if that's a very reproduceable speedup).

Really, the idea that actual values don't matter and only ordering does
is broken.  Of course actual values matter, because by how much
something is faster is a much more useful piece of information than
simply "it is faster".  If changing for another interpreter makes some
benchmark 3x faster, I may go for it.  If it makes some benchmark 3%
faster, I won't bother.

Regards

Antoine.