Hardware take on software testing.

[Moshe Zadka]

On Mon, 09 Jun 2003, Peter Hansen <peter at engcorp.com> wrote:

> We find we achieve 100% code coverage on code that is purely test-driven,
> and we _do_ expect no bugs.
<snip>
> though from time 
> to time we find them.  Make any sense?  I should have said "you should 
> expect no bugs", and didn't mean to imply you will never have them.

One area where you easily find lots and lots of bugs, even when you
have a good testing suite, is when porting software. Frequently, that's
irrelevant -- if you're developing an in-house solution which has to
run on, say, Debian Woody, and you're running tests on Debian Woody,
then everything is fine.

Where I saw this problem was in Twisted. Now, I do realize Twisted
has *far* from enough tests. But even the tests that do exist do not
keep us "bug free" as far as they go. For example, suppose I write
some code, put in a test, and it works. I happily commit. Now it
turns out that in Python 2.3, this test spits pending deprecation
warnings. No problems, I fix this, and test with both 2.2 and 2.3.
Now, the problem is that the test doesn't run if the machine needs
a proxy to connect to the web. OK, I hack around it. Now I run it
again, and it seems that my code had a race condition: to stop it
from hanging the test suite, I put in a time limit, but the limit
was too short. So, I increase the limit. Now I commit, and someone
running the unit tests with the qtreactor has problems. Having no
qtreactor-able set up, I cannot even be sure how to fix it.
So on, and on, ad neasuem. Now, I'm not saying that tests do not
help -- I am just saying that writing tests in portable ways,
even if the code they test is itself portable, is highly non-trivial

I predict the XP response to it: "define the platforms you want,
and then have the running-tests command automatically run on all
those platforms". Unfortunately, again real life intervenes. If every
commit had to run the test suite 10 times on five different machines,
little real work would be done.

-- 
Moshe Zadka -- http://moshez.org/
Buffy: I don't like you hanging out with someone that... short.
Riley: Yeah, a lot of young people nowadays are experimenting with shortness.
Agile Programming Language -- http://www.python.org/