Data-driven testing

[Dave Brueck]

On Wed, 23 Apr 2003, Aahz wrote:

> As I just mentioned in another post, I'm writing an ASCII database for
> my address book.  Actually, the main reason for writing it was because
> I've been maintaining multiple lists for various kinds of party invites,
> and the update problem was killing me.
>
> While writing it, it occurred to me that I *ought* to be using unit
> tests to make sure that it was working correctly, but two things stopped
> me.  Secondly, I was in a hurry.
>
> But firstly, I ran into the same problem I had when writing the BCD
> module: how the heck does one write tests when the results are
> hand-calculated?  There were several times when I wrote what I thought
> were correct tests for the BCD module, and my code was right and my test
> was wrong.

Hmm... seems like you'd have this problem regardless of whether or not
you write tests, right? In many ways your tests are just an automated
playback of an interactive debug/test session, so either way at some point
you have to bite the bullet and do a detailed check that the output is
right. Often you can intelligently pick a few good manual test cases that
exercize a good portion of the functionality and then do purely automated
testing of the rest (e.g. randomly choose a number, convert to BCD,
convert back, compare to original - that sort of thing).

Also, tests are a huge asset in that they detect regression in behavior,
so even if you have to double or triple check a few very manual tests it's
worth it because from then on they'll immediately notify you when
something has broken.

> In the case of my ASCII database, I'm going to need a big matrix to
> properly test the query language.  How do I build that matrix?

Exhaustive testing is sometimes economically infeasible unless you're a
government agency. :) If breaking the problem down and testing
subcomponents (in addition to end-to-end tests) still leaves too much,
then at least identify the most critical areas to test and focus on those.

The reason we don't all do formal verification of our code is that the
cost is too high. Likewise, in testing you test until the cost of a bug in
deployed code is lower than the cost of finding the bug through more
testing. For some applications it's worth it to test every code path,
every border case, etc. because it really is that critical. On the other
end of the spectrum (e.g. homegrown stuff where I'm my own customer) the
cost of a bug in "deployed" code is sometimes so small that I don't write
_any_ tests (I don't agree that writing tests _always_ saves you time -
although it _often_ or even _usually_ does).

-Dave