[Numpy-discussion] Performance testing in unit tests

[Gael Varoquaux]

On Thu, Aug 27, 2009 at 03:33:30PM -0700, Robert Kern wrote:
> From my experience, doing performance tests inside of your normal test
> suite is entirely unreliable. Performance testing requires rigorous
> control over external factors that you cannot do inside of your test
> suite. Your tests will fail when run in the entire test suite and pass
> when run by themselves, or vice versa.

> It can also be hard to run two similar tests serially thanks to any
> number of caches that might be in effect, but this is often
> manageable.

That is why it can be useful to repeat the measure several times, isn't
it?

> If you can manage that, then you can probably use nose or some other
> framework to conveniently run individually named tests in a reasonably
> controlled manner. There is not much unit test-specific to do, though.
> You time your two code paths and compare them inside of a
> test_function() just like you would do if you are writing an
> independent benchmark script.

OK, the following seems to be give quite reproducible results:

    # timing procedure:
    a = np.random.random(1000000)
    time_hash = list()
    for _ in range(3):
        t1 = time.time()
        hash(a)
        time_hash.append(time.time() - t1)
    time_hash = min(time_hash)
    
    time_hashlib = list()
    for _ in range(3):
        t1 = time.time()
        hashlib.md5(a).hexdigest()
        time_hashlib.append(time.time() - t1)
    time_hashlib = min(time_hashlib)
    relative_diff = abs(time_hashlib - time_hash)/time_hashlib
    nose.tools.assert_true(relative_diff < 0.05)

Thanks,

Gaël