[Python-Dev] Python Benchmarks

[Michael Chermside]

Marc-Andre Lemburg writes:
> Using the minimum looks like the way to go for calibration.
>
> I wonder whether the same is true for the actual tests; since
> you're looking for the expected run-time, the minimum may
> not necessarily be the choice.

No, you're not looking for the expected run-time. The expected
run-time is a function of the speed of the CPU, the architechure
of same, what else is running simultaneously -- perhaps even
what music you choose to listen to that day. It is NOT a
constant for a given piece of code, and is NOT what you are
looking for.

What you really want to do in benchmarking is to *compare* the
performance of two (or more) different pieces of code. You do,
of course, care about the real-world performance. So if you
had two algorithms and one ran twice as fast when there were no
context switches and 10 times slower when there was background
activity on the machine, then you'd want prefer the algorithm
that supports context switches. But that's not a realistic
situation. What is far more common is that you run one test
while listening to the Grateful Dead and another test while
listening to Bach, and that (plus other random factors and the
phase of the moon) causes one test to run faster than the
other.

Taking the minimum time clearly subtracts some noise, which is
a good thing when comparing performance for two or more pieces
of code. It fails to account for the distribution of times, so
if one piece of code occasionally gets lucky and takes far less
time then minimum time won't be a good choice... but it would
be tricky to design code that would be affected by the scheduler
in this fashion even if you were explicitly trying!


Later he continues:
> Tim thinks that it's better to use short running tests and
> an accurate timer, accepting the added noise and counting
> on the user making sure that the noise level is at a
> minimum.
>
> Since I like to give users the option of choosing for
> themselves, I'm going to make the choice of timer an
> option.

I'm generally a fan of giving programmers choices. However,
this is an area where we have demonstrated that even very
competent programmers often have misunderstandings (read this
thread for evidence!). So be very careful about giving such
a choice: the default behavior should be chosen by people
who think carefully about such things, and the documentation
on the option should give a good explanation of the tradeoffs
or at least a link to such an explanation.

-- Michael Chermside