[Speed] A new perf module: toolkit to write benchmarks

[Victor Stinner]

2016-06-02 9:17 GMT+02:00 Antoine Pitrou <solipsis at pitrou.net>:
> Just use a simple JSON format.

Yeah, Python 2.7 includes are JSON parser and JSON is human readble
(but not really designed to be modified by a human).

I had a technical issue: I wanted to produce JSON output *and* keep
nice human output at the same time. I found a nice trick: by default
write human output to stdout, but write JSON to stdout and human
output to stderr in JSON mode.

At the end, you get a simple CLI:
---
$ python3 -m perf.timeit --json 1+1 > run.json
.........................
Average: 18.3 ns +- 0.3 ns (25 runs x 3 samples x 10^7 loops)

$ python3 -m perf < run.json
Average: 18.3 ns +- 0.3 ns (25 runs x 3 samples x 10^7 loops)
---

The JSON can contain metadata as well:
---
$ python3 -m perf.timeit --metadata --json 1+1 > run.json
Metadata:
- aslr: enabled
- cpu_count: 4
- (...)
.........................
Average: 18.2 ns +- 0.0 ns (25 runs x 3 samples x 10^7 loops)

$ python3 -m perf < run.json
Metadata:
- aslr: enabled
- cpu_count: 4
- (...)
Average: 18.2 ns +- 0.0 ns (25 runs x 3 samples x 10^7 loops)
---

There are two kinds of objects: a single run, or a result composed of
multiple runs. The format is one JSON object per line.

Example of single runs using individual JSON files and then combine them:
---
$ python3 -m perf.timeit --raw --json 1+1 > run1.json
warmup 1: 18.3 ns
sample 1: 18.3 ns
sample 2: 18.3 ns
sample 3: 18.3 ns

$ python3 -m perf.timeit --raw --json 1+1 > run2.json
warmup 1: 18.2 ns
sample 1: 18.2 ns
sample 2: 18.2 ns
sample 3: 18.2 ns

$ python3 -m perf.timeit --raw --json 1+1 > run3.json
warmup 1: 18.2 ns
sample 1: 18.2 ns
sample 2: 18.2 ns
sample 3: 18.2 ns

$ python3 -m perf < run1.json   # single run
Average: 18.3 ns +- 0.0 ns (3 samples x 10^7 loops)

$ cat run1.json run2.json run3.json | python3 -m perf   # 3 runs
Average: 18.2 ns +- 0.0 ns (3 runs x 3 samples x 10^7 loops)
---

Victor