[issue23552] Have timeit warn about runs that are not independent of each other

[Robert Collins]

Robert Collins added the comment:

I think for PyPI its actually important here - the JIT'd state of the code is essentially global state being mutated - you can't assess how fast the code is without first warming up the JIT, and if it warms up half way through your fastest run, you're still not actually finding out what you might want to find out.

E.g. do you want to know:
 - how fast is this unjitted [e.g. CLI's]
 - how fast will this be once its hot [e.g. services]

Personally, I think as a first approximation, warning about massive variance is a good thing. We could add an option to turn it off, and we could also look at hooking properly into the jit to allow detection of stable state and benchmark only from there on in. But those extra things don't detract from the utility of warning about suspect runs.

----------
nosy: +rbcollins

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23552>
_______________________________________