[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations

[STINNER Victor]

STINNER Victor added the comment:

> With patch:
> python performance/bm_json_v2.py -n 10 --timer perf_counter

Oh. There is a bit of confusion here. You must *not* run directly bm_xxx.py scripts. The calibration is done in perf.py.

Try for example:
python perf.py python2 python3 -b json_dump_v2

You must see something like:
Calibrating => num_runs=10, num_loops=16 (0.50 sec < 0.55 sec)

I should maybe share the calibration code to also compute the number of iterations when a bm_xxx.py script is run directly? But the risk is that someone compares two runs of bm_xxx.py using two python binaries, and seeing different results just because the number of calibrated loops is different...

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26275>
_______________________________________