When is unit-testing bad? [was: Re: does lack of type...]

[Jeremy Bowers]

On Thu, 03 Jul 2003 01:14:44 +0000, Jeremy Bowers wrote:

> On Wed, 02 Jul 2003 15:21:45 +0200, Aurélien Géron wrote:
>> In all those cases, I much prefer having a human being go out and do the
>> unit tests manually.
> 
> The thing is, they won't. We've got about 50 years of collective
> experience backing that up.

It occurs I can expand on this. What I'm doing right now at my job (as
opposed to the posting I'm replying to which is my hobby), I'm writing
unittests for an existing mod_perl-based application that is several years
old now. Been an adventure.

Part of the reason I'm doing this is we're safely past the point where we
can just poke and stab at the system and get any sort of coverage. We've
got a role-based system (a user can have several types of roles, and
technically, it's a permission-based system where roles are bundles of
permissions...), and of course any number of things the user can do. Add
up 10-ish roles, any number of actions (lets just call it 50 for
simplicity, though there's no real way good way to put a number on
"distinct actions in this system"), and a couple of other dimensions, and
you've got a lot to cover. Well, actually, it's not addition, it's
multiplication, so even in this rather simple case, you've got 10*50*8 or
something things to cover.

Each of these actions are relatively easy to characterize as unit tests
(well, in proper XP terms these would be acceptance tests, which is still
a step up for the existing system and while I hope to see true unit tests
someday, a whole lotta refactoring would have to happen first), but we've
recently been getting *hammered* with bugs that only occur on one or two
cells in that (easily) 4000-cell matrix. (Of course they're the .5% fringe
cases, important enough to actually occur but not important enough to bang
on in poke-and-grunt testing.) We had a bug that only occurred with Role A
in situation B in environment-type C. Unfortunately, while none of us
ever though to test that exact combination of settings, it was reasonably
common in the real world. Fully automated testing with absolutely no human
interaction needed during the testing is really, really important,
and I can't believe it took me this long to realize that. Not to mention
the rest of the industry.

On that topic, I'm starting to develop a multi-level unit testing
philosophy, since it seems like every program I've ever worked on ends up
with these multi-dimensional tests, and *yes* the equivalent of (1,4,2,4)
will sometimes fail, all by itself. Is there any official XP dogma on
that? Running a complete suite would take too long to be practical, so I'm
starting to get this image of defining a subset of tests for development
purposes (each axis, selected diagonals, smattering of others), and
running the full suite, say, overnight for full coverage... or even
constantly on a spare machine. Would make a great demo for 
tourists coming into the lab, with suitably verbose output, probably
impress customers too. ;-)