Code correctness, and testing strategies

[Scott David Daniels]

David wrote:
> Specifically, if you've just written 100 new lines of Python code, then:
> 1) How do you test the new code?
> 2) How do you ensure that the code will work correctly in the future?
> 
> Short version:
> 
> For (1) I thoroughly (manually) test code as I write it, before
> checking in to version control.
> 
> For (2) I code defensively.
> 
...
> As for point 1 (how do you test the new code?):
> I like the idea of automated unit tests. However, in practice I find
> they take a long time to write and test, especially if you want to
> have good coverage (not just lines, but also possible logic branches).
This is why I have reluctantly come to accept the XP people's view:
if you you write the tests _as_ you develop (that is as far as I go
w/o re-enforcement; they would have you write them _before_), you will
have a body of tests that work to demonstrate the correctness or
deficiencies of your code based on what it _should_ do.  If you write
tests after you've written the code, you will write tests that are
based on what your code _actually_does_.  You don't want the latter;
the tests are brittle.  The tests don't match needs, rather they
match implementations.  Therefore you'll need to discard more tests at
every local rewrite.

> 1) Add "raise 'UNTESTED'" lines to the top of every function
String exceptions are deprecated.  Just raise UNTESTED (and let the
access to undefined global error be the issue).
...<describes how to do code coverage by hand>...
> 11) Cause those sections of code to be run also (sometimes I need to
> temporarily set vars to impossible values inside the script, since the
> logic will never run otherwise)
> 
> And here is one of my biggest problem with unit tests. How do you unit
> test code which almost never runs? The only easy way I can think of is
> for the code to have 'if <some almost impossible condition> or <busy
> running test case XYZ> lines'. I know I'm meant to make 'fake' testing
> classes which return erroneous values, and then pass these objects to
> the code being tested. But this can take a long time and even then
> isn't guaranteed to reach all your error-handling code.
Ah, but now you tests are "brittle"; they only work for the code you 
have now.
If you want to make sure you have code coverage with your test, the XP
way is:
     Write a test for behavior you need.
     Watch it fail.
     Fix the code so all tests pass.
     Lather, rinse, repeat.
You should not have untested code, because there was no test that made
you write it.  If you want to do code coverage, find a code coverage
tool and count your code while runnign your unit tests.


--Scott David Daniels
Scott.Daniels at Acm.Org