Test professionalism (was: count items in generator)

[Delaney, Timothy (Tim)]

Cameron Laird wrote:

> Gulp.  OK, you've got me curious:  how many people habitually frame
> their unit tests with resource constraints?  I think I take testing
> seriously, and certainly also am involved with resource limits often,
> but I confess I've never aimed to write all my tests in terms of
> bounds on time (and presumably memory and ...).  You've got me
> thinking, Tim.

Generally we only do it when we actually discover a problem i.e. the
test has been running forever ... which is precisely the problem. At
least you don't have to specifically code tests for bounds on most other
resources (such as memory) as eventually it will fail with a language
with managed memory (Python, Java, etc). It definitely becomes an issue
for C/C++ unit tests though - they don't fail nicely.

I believe JUnit 4 includes the option to time out any test, but that you
can vary the timeout using an annotation (for example, make any test
fail if it takes longer than 30 seconds - but you may have one test that
needs to run longer). We're only using JUnit 3 at my work at the moment
though.

The application I'm currently working on is very timing dependent (and
the timings vary greatly across machines), so we've got loops like:

    while (test condition not met) and (timeout not exceeded):
        sleep

    assert test condition met

Originally these were just:

    while (test condition not met):
        sleep

and as a result turned into infinite loops.

I think it's much better to have every test have a default timeout (at
which point the test fails) and be able to change that default when
needed. This protects against errors in both the code, and test cases.

Tim Delaney