Code correctness, and testing strategies

[D'Arcy J.M. Cain]

On Sat, 24 May 2008 21:14:36 +0200
David <wizzardx at gmail.com> wrote:
> Is it considered to be cheating if you make a test case which always
> fails with a "TODO: Make a proper test case" message?

Yes.  It's better to have the daily reminder that some code needs to be
finished.

> While it is possible to describe all problems in docs, it can be very
> hard to write actual test code.

It may be hard to start but once you have your framework in place it
becomes very easy.

> For example: sanity tests. Functions can have tests for situations
> that can never occur, or are very hard to reproduce. How do you unit
> test for those?

Believe me, thousands of people reading this are remembering situations
where something that couldn't possibly happen happened.

> A few examples off the top of my head:
> 
> * Code which checks for hardware defects (pentium floating point,
> memory or disk errors, etc).
> 
> * Code that checks that a file is less than 1 TB large (but you only
> have 320 GB harddrives in your testing environment).
> 
> * Code which checks if the machine was rebooted over a year ago.
> 
> And so on. These I would manually test by temporarily changing
> variables in the code, then changing them back. To unit test these you
> would need to write mock functions and arrange for the tested code to
> call them instead of the python built-ins.

Yes but the mock functions can be wrappers around the real functions
which only change the results that you are testing for.

> eg: You call function MyFunc with argument X, and expect to get result Y.
> 
> MyFunc calls __private_func1, and __private_func2.
> 
> You can check in your unit test that MyFunc returns result Y, but you
> shouldn't check __private_func1 and __private_func2 directly, even if
> they really should be tested (maybe they sometimes have unwanted side
> effects unrelated to MyFunc's return value).

It isn't your job to test __private_func1 and __private_func2 unless
you are writing MyFunc.

> Depends on the type of bug. If it's a bug which breaks the unit tests,
> then it can be found quickly. Unit tests won't help with bugs they
> don't explicitly cover. eg off-by-one, memory leaks, CPU load,
> side-effects (outside what the unit tests test), and so on.

No but when you find that your code breaks due to these problems that's
when you write new unit tests.

> But once you track down problems like the above you can write more
> unit tests to catch those exact bugs in the future. This is one case
> where I do favour unit tests.

Yes!  One of the biggest advantages to unit testing is that you never
ever deliver the same bug to the client twice.  Delivering software
with a bug is bad but delivering it with the same bug after it was
reported and fixed is calamitous.

-- 
D'Arcy J.M. Cain <darcy at druid.net>         |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 425 1212     (DoD#0082)    (eNTP)   |  what's for dinner.