Testing for performance regressions

[Steven D'Aprano]

I'm writing some tests to check for performance regressions (i.e. you 
change a function, and it becomes much slower) and I was hoping for some 
guidelines or hints.

This is what I have come up with so far:


* The disclaimers about timing code snippets that can be found in the 
timeit module apply. If possible, use timeit rather than roll-you-own 
timers.

* Put performance tests in a separate test suite, because they're 
logically independent of regression tests and functional tests, and 
therefore you might not want to run them all the time.

* Never compare the speed of a function to some fixed amount of time, 
since that will depend on the hardware you are running on, but compare it 
relative to some other function's running time. E.g.:

# Don't do this:
time_taken = Timer(my_func).timeit()  # or similar
assert time_taken <= 10
    # This is bad, since the test is hardware dependent, and a change 
    # in environment may cause this to fail even if the function 
    # hasn't changed.

# Instead do this:
time_taken = Timer(my_func).timeit()
baseline = Timer(simple_func).timeit()
assert time_taken <= 2*baseline
    # my_func shouldn't be more than twice as expensive as simple_func
    # no matter how fast or slow they are in absolute terms.


Any other lessons or hints I should know?

If it helps, my code will be targeting Python 3.1, and I'm using a 
combination of doctest and unittest for the tests.


Thanks in advance,



-- 
Steven.