Merits of otherwise of test-first (Re: connect four (game))

[nospam.Gregory Ewing]

bartc wrote:
> Testing everything comprehensively just wouldn't be useful for me who
> works on whole applications, whole concepts, not just a handful of
> functions with well-defined inputs and outputs.

I had this experience with Pyrex (the precursor to Cython). The various parts
are so interdependent that it doesn't really make sense to test individual
classes or methods on their own. It would require a huge amount of scaffolding
and wouldn't really prove much.

Instead, what I did was write tests for units of *functionality*. (Can it
compile an assignment statement? An expression that adds two things? Subtracts
them? etc.) I wrote a very large number of them, plus a test framework to run
them and compare the output with known good output.

I can't claim that I always wrote the tests *first* (sometimes I did, sometimes
 I didn't). But I never had any trouble motivating myself to write the tests.
Partly this is because the way my test framework worked, there was very little
difference between a manual test and an automated one. I had to write a test
file to test it at all, and having done that, it was just as easy to save it
away together with its output so that it became part of the automated test
suite.

It was also partly because the benefits of having a comprehensive automatic
test suite were so blindingly obvious in that project. Its complexity meant
that it was particularly easy to break something accidentally when adding a new
 feature. If that happened, the tests would almost certainly pick it up
immediately. Without those tests, I would never have been able to develop it to
 the point I did.

>> So there was something else wrong with the chip. I'm not sure what
>> your point is.
>
> The extensive testing was like unit testing, but needed to be even more
> thorough because of the commitment involved. It failed to spot a problem.

Testing obviously isn't guaranteed to find all problems in a system with
complex behaviour, but that doesn't mean testing is useless. It would have been
 hugely more risky to go ahead and build the chip without testing the design at
 all!

BTW, hardware is a bit different from software. It's possible
for a design to be *logically* correct and pass all the simulations, but for
the hardware to fail for messy physics- related reasons. (Charles Moore tells a
 story like that about one of his Forth chips. A transistor driving a
particular signal was too small, and if you executed a particular instruction
too many times in too short a time interval, it would overheat and produce
incorrect results.)

--
Greg