Code correctness, and testing strategies

[Duncan Booth]

David <wizzardx at gmail.com> wrote:

> So, at what point do you start writing unit tests? Do you decide:
> "Version 1 I am going to definitely throw away and not put it into
> production, but version 2 will definitely go into production, so I
> will start it with TDD?".

If you are going to prototype something without tests then you decide that 
up-front and throw the code away afterwards. It never is version 1 nor 2.

> 
> Where this doesn't work so well is if version 2 is a refactored and
> incrementally-improved version of version 1. At some point you need to
> decide "this is close to the version that will be in production, so
> let's go back and write unit tests for all the existing code".

No you've gone beyond prototyping at that point. Prototypes are for 
throwing away.

> Seriously, 10 hours of testing for code developed in 10 hours? What
> kind of environment do you write code for? This may be practical for
> large companies with hordes of full-time testing & QA staff, but not
> for small companies with just a handful of developers (and where you
> need to borrow somone from their regular job to do non-developer
> testing). In a small company, programmers do the lions share of
> testing. For programmers to spend 2 weeks on a project, and then
> another 2 weeks testing it is not very practical when they have more
> than one project.

I've done plenty of projects using conventional methods in small companies. 
If anything test time equal to development time is a vast underestimate.

> 
> As for your other points - agreed, bugs getting to the customer is not
> a good thing. But depending on various factors, it may not be the end
> of the world if they do. eg: There are many thousands of bugs in open
> source bug trackers, but people still use open source for important
> things. Sometimes it is better to have software with a few bugs, than
> no software (or very expensive, or very long time in development). See
> "Worse is Better": http://en.wikipedia.org/wiki/Worse_is_better. See
> also: Microsoft ;-)
> 

I'm not saying it is the end of the world. What it is though is expensive. 
If as the developer you spot a problem and fix it then all the time is 
yours. If it is caught in QA then it involves bug reporting, assigning 
priority, fixing and then the code has to be re-tested. If the customer 
finds the problem then customer support has to report it, someone has to 
assign it a priority, someone has to figure out how to replicate it, then 
someone has to figure out the problem, then you have to fix it, then the 
code has to be re-tested. Every level out from the initial developer you 
increase the number of people and number of steps involved.

The other reason is that if your unit test breaks after you've just written 
5 lines of code then you know which 5 lines of code are the problem. 
Probably you can see the problem instantly. At worst you just hit undo in 
your editor and try again. If a bug report comes back months later you have 
no idea where look in thousands of lines of code.

> 
> The next time your project is running late, your manager and the
> customer will be upset if you spend time updating your unit tests
> rather than finishing off the project (and handing it over to QA etc)
> and adding the unit tests when there's actually time for it.

But you aren't updating unit tests. What you are doing is coding: the tests 
are an integral part of that, or rather the tests are part of the final 
design rather than coding.

> 
>>> Clients, deadlines, etc require actual software, not
>>> tests for software (that couldn't be completed on time because you
>>> spent too much time writing tests first ;-)).
>>
>> Clients generally require *working* software. Unfortunately it is all
>> too easy to ship something broken because then you can claim you
>> completed the coding on time and any slippage gets lost in the next 5
>> years of maintenance.
> 
> That's why you have human testing & QA. Unit tests can help, but they
> are a poor substitute. If the customer is happy with the first
> version, you can improve it, fix bugs, and add more unit tests later.

No, they aren't a substitute at all. You still need human testing and QA, 
the difference is that with a good set of unit tests you reduce the number 
of times code comes back from QA before it can be passed and make it more 
likely that the customer will be happy with the first version.