[Speed] [Core-mentorship] The Grand Unified Python Benchmark Suite

[Mark Shannon]

Antoine Pitrou wrote:

[snip]

> 
> By the way, the Mako benchmark shows a worrying regression (3x slower)
> on your new dict implementation.

Take a look at the timeline graph, is it very noisy?

There is a flaw in the benchmarking code in that the runs are not 
interleaved, so other processes tend to introduce systematic errors.

For example, here is a run of mako (comparing my dict with tip):

### mako ###
Min: 0.805583 -> 0.839515: 1.04x slower
Avg: 0.831936 -> 0.910184: 1.09x slower
Significant (t=-3.25)
Stddev: 0.01302 -> 0.11953: 9.1820x larger

It is 9% slower, right?
Wrong. Take a look at the timeline:
http://tinyurl.com/82l9jna
its 1-2% slower, but another process grabs the CPU for some of the 
iterations.

This should not be a problem for speed.python.org as it will have a
dedicated machine, but you need to be careful when benchmarking on your
desktop machine.

As an experiment, try benchmarking a python build against itself and see 
what you get.

Cheers,
Mark.