When is unit-testing bad? [was: Re: does lack of type...]

[John J. Lee]

Donn Cave <donn at u.washington.edu> writes:

> In article <3EEE7D4C.DA0B3512 at engcorp.com>,
[...]
> He's not the only one who gets a message something like that
> from the unit testing advocates on c.l.p.  Without the hyperbolic
> ``all possible situations'', sure, but more to the point, does
> the discussion usually acknowledge significant exceptions, or
> rather tend to dismiss them?

Nobody can rationally disagree with you that there might exist
situations where writing tests is inefficient (saves less time than it
costs).  But, of course, we'd like to know *what those cases are*,
because, AFAIK, they haven't been clearly pointed out.

I assume you have no problem accepting that many people have found
that writing lots of automated unit tests, and writing them early,
works better than not doing so.

So, what criteria do you have for the test/no-test decision that work
better than my own rule of "always write an automated unit test
(probably before writing the code), unless I'm too lazy or
incompetent" <0.2 wink>? (lazy in the bad sense, that is)

I suppose people might argue that if the answers to the following
questions, for example (any others to suggest?), are mostly 'no', then
the ratio of tests to code should be lower:

1. Code likely to be reused in future?
2. Code has been reused in past?
3. Code will not grow big (say, > 1000 lines)?
4. Lots of people working on it?
5. For a long time?
6. Correctness of code particularly important?
7. Emphasis on refactoring?


But you're probably already expecting the answers:

1. Often hard to tell if it will be.
2. But adding tests later is more expensive (as is fixing bugs later).
3. See 1, 2.
4. See 1, 2.  But even if it's only one person, it's usually worth it (IME).
5. See 1, 2.
6. I have to admit code correctness is *not* always crucial (whatever
   Bertrand Meyer may try to tell you), but I suspect that, usually, if
   it's important enough to fix bugs, it's worth writing tests.
7. My mind isn't entirely made up, but it might be suggested that if
   you're not refactoring, you've made another mistake.


Looking at the thread from which I'm trying to start this one, a
lesson to be learned from the static vs. dynamic typing thing is that
it doesn't always work to consider one thing at a time.  Static-typing
wonks like to point out that static analysis finds bugs.  But us
dynamic typing, um, types, note that static analysis isn't the only
variable: dynamic typing brings other pros, static typing other cons,
and (dynamic typing + tests) is better than the other combinations.
So maybe unit-testing enthusiasts should be thinking more about the
costs and opportunity costs of testing?  Maybe if we all switched to
constraint-based / functional languages with static type inference and
god-knows-what-else, and spent some of our testing time doing more
code review, we'd be better off still?


John