[Speed] A new perf module: toolkit to write benchmarks

[Victor Stinner]

2016-06-02 10:38 GMT+02:00 Armin Rigo <arigo at tunes.org>:
> Looks good.  IMHO the important bit is that `timeit` is simple to use,
> readily available, and gives just a number, which makes it very
> attractive for people.

By default min & max are hidden. You can show them using -v option.

To make the output even simpler, maybe the standard deviation can be
displayed "in english". Something like:

* "Average: 250 ns +- 3 ns" => "Average: 250 ns (stable)", or just
"Average: 250 ns"
* "Average: 250 ns +- 120 ns" => "Average: 250 ns (not reliable, try
again on an idle system)"

Usually, timeit it used to compare two versions of Python. Maybe we
should focus on this use case, and check if the difference is
significant, as perf.py does? By default, perf.py does *not* display
any number if the difference is not significant. I like this
behaviour, even if it can be surprising for the first time.

For the CLI, we can extend timeit CLI to accept the path/name of two
python binaries. Or we can use something like pybench to store result
into files and then load & compare two files.

Victor