[Speed] Experiences with Microbenchmarking

[Edd Barrett]

Hi,

A colleague has just pointed me to the discussions on this list
regarding benchmarking methodology. Over the past few months we have
been devising an "as rigorous as possible" micro-benchmarking
experiment. It seems there's a lot of crossover in our work and your
discussions.

In short, our experiment is investigating the warmup behaviours of
JITted VMs (currently PyPy, HotSpot, Graal, LuaJIT, HHVM, JRubyTruffle
and V8) using microbenchmarks. For each microbenchmark/VM pairing we
sequentially run a number of processes (currently 10), and within each
process we run 2000 iterations of the microbenchmark. We then plot the
results and make observations.

The experiments were run under our own "paranoid" benchmark runner
(Krun), which aims to control as many confounding variables as are
practically possible. Amongst others, it checks that all benchmarks are
run with the system at a similar starting temperature, disables ASLR,
uses a monotonic system clock (in some cases we had to patch VMs) and it
reboots the system before each benchmark. We did not isolate CPUs, since
we found that this creates artificial contention on multi-threaded VMs,
however, we did use (and Krun checks for) a tickless Linux kernel.

We expected to see typical warmup behaviours (with distinct phases for
profiling, compilation, and peak performance), but in reality we saw
all kinds of crazy behaviours and even slowdowns.

We've published a draft paper showing our preliminary findings here:
http://arxiv.org/abs/1602.00602

The draft shows a subset of our results. Run-sequence plots for all
process executions can be found here:
https://archive.org/download/softdev_warmup_experiment_artefacts/v0.1/all_graphs.pdf

For the final version of the paper we are trying to devise statistical
methods to automatically classify the strange warmup behaviours we
encountered. We will also run CPython in our final experiment, which may
interest you guys :)

If this interests anyone, I'd be happy to discuss further.

Cheers

-- 
Best Regards
Edd Barrett

http://www.theunixzoo.co.uk