[Speed] perf 0.6 released

[Victor Stinner]

Hi,

I'm pleased to announce the release of the Python perf modul version 0.6.

The main difference is that the JSON format and perf command line
tools support benchmark suites and not only individual benchmarks.

I added a new "convert" command which allows to modify a benchmark
file: remove benchmarks, remove benchmarks runs, and a special "remove
outliers" command. I'm not sure that removing outliers is a good
practice, I will have to play with it to give you a feedback :-)

I added --fast and --rigorous options: simple options to configure the
number of processes and the of sample per process. The idea of these
options comes from the CPython benchmark suite.

I added --hist and --stats options to TextRunner, so it's now possible
to directly render an histogram and compute statistics on a benchmark
(without having to use a file).

Finally, the --json-append option allows to append a benchmark to an
existing benchmark suite file. It allows to "concatenate" multiple
benchmarks into a single JSON file.


timeit example showing new features:
---
$ python3 -m perf timeit -s 'x=" abc"' 'x.strip()' --stats --hist -v
--rigorous --json=timeit.json
calibration: 1 loop: 2.40 us
calibration: 2 loops: 1.60 us
(...)
calibration: 2^20 loops: 118 ms
calibration: use 2^20 loops
Run 1/20: warmup (1): 116 ms; raw samples (5): 116 ms, 118 ms, 116 ms,
116 ms, 136 ms (+17%)
Run 2/20: warmup (1): 119 ms; raw samples (5): 115 ms, 116 ms, 121 ms,
118 ms, 128 ms (+9%)
Run 3/20: warmup (1): 272 ms; raw samples (5): 208 ms (+76%), 117 ms,
121 ms, 122 ms, 119 ms
(...)
Run 20/20: warmup (1): 140 ms; raw samples (5): 116 ms, 115 ms, 115
ms, 116 ms, 115 ms

106 ns: 38 #################################
111 ns: 44 ######################################
115 ns:  8 #######
119 ns:  2 ##
124 ns:  5 ####
128 ns:  2 ##
133 ns:  0 |
137 ns:  0 |
141 ns:  0 |
146 ns:  0 |
150 ns:  0 |
155 ns:  0 |
159 ns:  0 |
164 ns:  0 |
168 ns:  0 |
172 ns:  0 |
177 ns:  0 |
181 ns:  0 |
186 ns:  0 |
190 ns:  0 |
195 ns:  1 #

Number of samples: 100 (20 runs x 5 samples; 1 warmup)
Loop iterations per sample: 2^20
Raw sample minimum: 115 ms
Raw sample maximum: 208 ms

Minimum: 110 ns (-1%)
Median +- std dev: 111 ns +- 10 ns
Mean +- std dev: 114 ns +- 10 ns
Maximum: 198 ns (+79%)

Median +- std dev: 111 ns +- 10 ns
---

The list of runs now highlight outliers by showing the percent for
samples out of the range [median - 5%; median + 5%]. Example: "raw
samples (5): 208 ms (+76%)".


Example of removing outliers:
---
$ python3 -m perf convert timeit.json --remove-outliers -o timeit2.json

haypo at selma$ python3 -m perf show --hist --stats -v timeit2.json
Run 1/12: warmup (1): 124 ms; raw samples (5): 117 ms, 119 ms, 118 ms,
117 ms, 118 ms
Run 2/12: warmup (1): 119 ms; raw samples (5): 119 ms, 117 ms, 117 ms,
118 ms, 117 ms
Run 3/12: warmup (1): 116 ms; raw samples (5): 117 ms, 116 ms, 116 ms,
117 ms, 118 ms
(...)
Run 12/12: warmup (1): 140 ms; raw samples (5): 116 ms, 115 ms, 115
ms, 116 ms, 115 ms

110 ns: 10 ###########################
110 ns: 14 ######################################
110 ns: 10 ###########################
111 ns:  7 ###################
111 ns:  2 #####
111 ns:  1 ###
112 ns:  5 ##############
112 ns:  3 ########
112 ns:  1 ###
112 ns:  1 ###
113 ns:  3 ########
113 ns:  0 |
113 ns:  1 ###
114 ns:  1 ###
114 ns:  0 |
114 ns:  0 |
115 ns:  0 |
115 ns:  0 |
115 ns:  0 |
115 ns:  0 |
116 ns:  1 ###

Number of samples: 60 (12 runs x 5 samples; 1 warmup)
Loop iterations per sample: 2^20
Raw sample minimum: 115 ms
Raw sample maximum: 121 ms

Minimum: 110 ns (-1%)
Median +- std dev: 111 ns +- 1 ns
Mean +- std dev: 111 ns +- 1 ns
Maximum: 116 ns (+5%)

Median +- std dev: 111 ns +- 1 ns
---

Without outliers, the histogram "looks better" but it changed a lot
the standard deviation (11 ns => 1 ns).

Victor