[Speed] steps to get pypy benchmarks running

[Sarah Mount]

On Tue, Feb 13, 2018 at 2:27 PM, Matti Picus <matti.picus at gmail.com> wrote:
>
> I have begun to dive into the performance/perf code. My goal is to get pypy benchmarks running on http://speed.python.org. Since PyPy has a JIT, the benchmark runs must have a warmup stage. There are some first-cut warmup values hardcoded inside a few of the benchmarks. I think I would prefer to use a different mechanism - a separate calibrated data file alongside the performance benchmarks. We could start off with a rough guess for the benchmarks and get the system up and running, and then calibrate the warmups hopefully finding some statistical basis for the values.
>

Unfortunately, it is very difficult to determine when warmup has
occurred, even on a per-vm-per-benchmark basis. If you allow
benchmarks to run for long enough, some will fail to even reach a
steady state where warmup can be said to have finished.

We have a more detailed paper on this which might interest you:

http://soft-dev.org/pubs/html/barrett_bolz-tereick_killick_mount_tratt__virtual_machine_warmup_blows_hot_and_cold_v6/

and some software which can do something similar to what you propose
-- i.e. determine whether and when warmup has occurred:

http://soft-dev.org/src/warmup_stats/

this includes a script which can "diff" benchmarks between (say)
commits, or between different versions of a VM. There's an example
table on the link above, just hover over the table to see the diff.

If you want to have a go at using this, feel free to get in touch --
we'd be happy to help!

Regards,

Sarah

-- 
Dr. Sarah Mount, Research Associate, King's College London
Fellow of the Software Sustainability Institute
twitter: @snim2