[Numpy-discussion] Performance testing in unit tests

[Robert Kern]

On Thu, Aug 27, 2009 at 15:03, Gael
Varoquaux<gael.varoquaux at normalesup.org> wrote:
> Hi list,
>
> This is slightly off topic, so please pardon me.
>
> I want to do performance testing. To be precise, I have a simple case: I
> want to check that 2 operations perform with a similar speed (so I am
> abstracted from the machines performance).
>
> What would be the recommended way of timing the operation in a unit test?
> I use nose, if this is of any use. I am more than happy to be pointed to
> an example.

>From my experience, doing performance tests inside of your normal test
suite is entirely unreliable. Performance testing requires rigorous
control over external factors that you cannot do inside of your test
suite. Your tests will fail when run in the entire test suite and pass
when run by themselves, or vice versa.

It can also be hard to run two similar tests serially thanks to any
number of caches that might be in effect, but this is often
manageable. If you can manage that, then you can probably use nose or
some other framework to conveniently run individually named tests in a
reasonably controlled manner. There is not much unit test-specific to
do, though. You time your two code paths and compare them inside of a
test_function() just like you would do if you are writing an
independent benchmark script.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco