Merits of otherwise of test-first (Re: connect four (game))

[Gregory Ewing]

bartc wrote:
> Testing everything comprehensively just wouldn't be useful for me who 
> works on whole applications, whole concepts, not just a handful of 
> functions with well-defined inputs and outputs.

I had this experience with Pyrex (the precursor to Cython).
The various parts are so interdependent that it doesn't really
make sense to test individual classes or methods on their own.
It would require a huge amount of scaffolding and wouldn't
really prove much.

Instead, what I did was write tests for units of *functionality*.
(Can it compile an assignment statement? An expression that adds
two things? Subtracts them? etc.) I wrote a very large number of
them, plus a test framework to run them and compare the output
with known good output.

I can't claim that I always wrote the tests *first* (sometimes
I did, sometimes I didn't). But I never had any trouble motivating
myself to write the tests. Partly this is because the way my
test framework worked, there was very little difference between
a manual test and an automated one. I had to write a test file
to test it at all, and having done that, it was just as easy
to save it away together with its output so that it became
part of the automated test suite.

It was also partly because the benefits of having a comprehensive
automatic test suite were so blindingly obvious in that project.
Its complexity meant that it was particularly easy to break
something accidentally when adding a new feature. If that
happened, the tests would almost certainly pick it up immediately.
Without those tests, I would never have been able to develop
it to the point I did.

>> So there was something else wrong with the chip. I'm not sure what
>> your point is.
> 
> The extensive testing was like unit testing, but needed to be even more 
> thorough because of the commitment involved. It failed to spot a problem.

Testing obviously isn't guaranteed to find all problems in
a system with complex behaviour, but that doesn't mean
testing is useless. It would have been hugely more risky to
go ahead and build the chip without testing the design at
all!

BTW, hardware is a bit different from software. It's possible
for a design to be *logically* correct and pass all the
simulations, but for the hardware to fail for messy physics-
related reasons. (Charles Moore tells a story like that about
one of his Forth chips. A transistor driving a particular
signal was too small, and if you executed a particular
instruction too many times in too short a time interval,
it would overheat and produce incorrect results.)

-- 
Greg