Code correctness, and testing strategies

[Ben Finney]

David <wizzardx at gmail.com> writes:

> Problem 1: You can only code against tests
> 
> Basically, with TDD you write the tests first, then the code which
> passes/fails the tests as appropriate. However, as you're writing
> the code you will also think of a lot of corner cases you should
> also handle. The natural way to do this is to add them to the code
> first.

In Behaviour Driven Development, that's not the natural way, since you
don't add code unless a test requires you to add it.

Very often, by designing the code to the point where you write an
automated test for it, you find You Aren't Gonna Need It
<URL:http://c2.com/cgi/wiki?YouArentGonnaNeedIt> for many of the
corner cases you envisaged, so you *save* time by thinking about the
code design first.

> But with TDD you have to first write a test for the corner case,
> even if setting up test code for it is very complicated.

For this, unit test frameworks have "fixtures": a common test
environment that will be set up before each test case that is defined
to need that fixture, and torn down after the test runs (whether the
test result is pass, fail, or error).

In Python's 'unittest', this is done by grouping related test cases
together into methods of a TestCase class, and defining fixtures that
will be present for every test case in that class.

> So, you have these options:
> 
> - Take as much time as needed to put a complicated test case in
> place.

Which is a motivator to make the interfaces less complicated, *before*
implementing them, so the code becomes easier to test.

> - Don't add corner case to your code because you can't (don't have
> time to) write a test for it.

That's a false economy.

If you don't have the time to write a test for it, that's the same
thing as saying you don't have the time to know whether you've done
the implementation right.

You pay for it later, with *much* interest, in time spent hunting down
bugs caused by that code, when you could have had an automated test
simply tell you the breakage and its location as soon as it happened.
That's time that you spend before even getting to the point of fixing
the bug. Thus, writing tests first *saves* time, by not having to hunt
those bugs that could be found by automated tests at all.

> - Add the corner case handling to the code first, and try to add a
> test later if you have time for it.

A recipe for disaster: now you have code which isn't there to satisfy
a test, and since you say it's for a corner case, it's likely not
exercised by the test suite at all. Take everything said above about
debugging time and multiply it.

> Problem 2: Slows down prototyping
> 
> In order to get a new system working, it's nice to be able to throw
> together a set of modules quickly, and if that doesn't work, scrap
> it and try something else. There's a rule (forget where) that your
> first system will always be a prototype, regardless of intent.

Fred Brooks gave that law. "Plan to throw one away; you will, anyhow."

The same rule also implies that throwing it away is worthwhile because
the main value of the first version is not in the code itself, but in
what it *taught* you about how to make the second version.

> With TDD, you have to first write the tests for the first version.
> Then when that first version doesn't work out, you end up scrapping
> the tests and the code. The time spent writing the tests was wasted.

I find that, with a good unit test suite in place, the first version
can much more easily *evolve* into the second version: the first
version gets thrown away, but in pieces over a short time, instead of
a whole lump.

Complete coverage by unit tests makes bold changes to the code far
less risky, so as enlightenment dawns about how crappy the original
design was, altering the existing system is much more natural.

> Having to write tests for all code takes time.

Yes. That's time you'll have to spend anyway, if you want your code
tested. Whether you write the tests, early or late, they'll still take
time to write. But writing them *while* you develop the code means you
actually *do* write them, instead of putting them off to the point
where they never get written.

also, since with Behaviour Driven Development you're writing tests at
the point where you implement the code, you are writing *only* tests
that are needed to assert behaviour for the code you're about to
write. This results in countless wrong paths not being taken as a
result of smaller feedback cycles, so you save time in all the
unnecessary tests you *don't* write.

If you don't want your code to have complete automated unit test
suites, I don't want to be your customer.

> Instead of eg: 10 hours coding and say 1/2 an hour manual testing,
> you spend eg: 2-3 hours writing all the tests, and 10 on the code.

This is a false economy. If all you do is 2-3 hours of manual testing,
are you then prepared to say the code works for *all* requirements
with a high degree of confidence?

As I'm sure you'll agree, automated, complete-coverage testing for all
code paths is far superior confidence to 2-3 hours of manual testing.

> Problem 4: Can make refactoring difficult.

Your explanation for this below is only that it takes time, not that
it's difficult to do.

> If you have very complete & detailed tests for your project, but one
> day you need to change the logic fundamentally (maybe change it from
> single-threaded to multi-threaded, or from running on 1 server to
> distributed), then you need to do a large amount of test refactoring
> also.

Again, this is not a problem with the *way* the tests are written, nor
*when* they're written, but the fact that they're there at all.

> The more tests you have (usually a good thing), the longer it will
> take to update all the tests, write new ones, etc. It's worse if you
> have to do all this first before you can start updating the code.

Who says you have to do all the changes to all the tests before
changing any of the code? That would be spectacularly bad practice.

Behaviour Driven Development says that you address *one* change in
behaviour at a time, so that you know which failing tests to address.
This makes the work much more manageable, since it localises the
changes to both the tests and the application.

> You need to justify the extra time spent on writing test code.

>From the perspective of someone who once thought this way, and then
dug in and did Behaviour Driven Development, I can tell you the time
is entirely justified: by better design, more maintainable code,
higher confidence when speaking to customers, high visibility of
progress toward completion, freedom to refactor code when it needs it,
and many other benefits.

> Tests are nice, and good to have for code maintainability, but they
> aren't an essential feature (unless you're writing critical software
> for life support, etc).

Whether your customers express it at the time they specify the system,
they want as much testing as you can fit in. Whom do you think they
will blame when the application breaks?

If their specified requirements explicitly describe some behaviour,
but you don't have automated tests to tell you whether or not the
application actually behaves that way, and the application breaks when
they try it after paying you to implement it, what possible excuse do
you think the customers will accept?

> Clients, deadlines, etc require actual software, not tests for
> software (that couldn't be completed on time because you spent too
> much time writing tests first ;-)).

Clients also require meaningful feedback on progress toward the goal.

Without a complete-coverage unit test suite, you have a completely
unknown amount of debugging time looming that grows the longer the
project continues. Any figure of completion that you give to the
customer will be wildly inaccurate because of this unknown factor, and
it will bits when you get to the point of "90% feature complete", but
the project blows out hugely because completing that last 10% involves
masses of debugging time.

You've no doubt heard the old saw about software project development:
90% of the implementation consumes 90% of the schedule. The last 10%
consumes another 90% of the schedule."

With an automated unit test suite that has complete coverage of the
code at all stages of implementation, you can be much more confident
that the features *are* complete as specified, because the
specifications are codified as unit tests, and the unit tests pass.
You're checking items off a meaningful list, rather than coding and
praying that you got it right.

> I think that automated tests can be very valuable for
> maintainability, making sure that you or other devs don't break
> something down the line. But these benefits must be worth the time
> (and general inconvenience) spent on adding/maintaining the tests.

I encourage you to try it. Learn your unit test suite (Python's
'unittest' and 'doctest' modules are fine), learn how to easily run it
automatically (the third-party 'nose' package is great for this), and
follow Behaviour Driven Development on one small aspect of your code;
perhaps a single module.

> If I did start doing some kind of TDD, it would be more of the
> 'smoke test' variety. Call all of the functions with various
> parameters, test some common scenarios, all the 'low hanging fruit'.
> But don't spend a lot of time trying to test all possible scenarios
> and corner cases, 100% coverage, etc, unless I have enough time for
> it.

This was my point of view too. Then I discovered that the code I was
writing under Behaviour Driven Development involved far fewer
dead-ends, far better designs, and far more constant forward progress
— all as a result of the fact that I had to design the code well
enough to write tests for it, *before* actually implementing that
code.

> I'm going to read more on the subject (thanks to Ben for the link).
> Maybe I have some misconceptions.

If so, you're certainly not alone.

Happy hacking!

-- 
 \              “You can be a victor without having victims.” —Harriet |
  `\                                                             Woods |
_o__)                                                                  |
Ben Finney