[Tutor] IDEs

[Steven D'Aprano]

Alan Gauld wrote:

> The basic idea in testing is to try to break your code. Try to think
> of every kind of evil input that could possibly come along and see
> if your code survives. In amongst all of that you should have a
> some valid values too, and know what to expect as out put.

Testing is more than just trying to break code of course. I'm reminded 
of a quote:

"I find it amusing when novice programmers believe their main job is
preventing programs from crashing. ... More experienced programmers
realize that correct code is great, code that crashes could use
improvement, but incorrect code that doesn't crash is a horrible
nightmare." -- Chris Smith

Testing is an attempt to ensure that code is *correct* -- that it does 
what it is supposed to do.


There are a number of different types of tests, with different purposes.

Unit tests are for testing that code behaves as expected. If you give 
the function this input, this will be its result. (The result might be 
to return a value, or it might be to raise an exception.) If you have a 
function that's supposed to add two numbers, it's important to be sure 
that it actually, ya know, *adds two numbers*, and not something else. 
("It *almost* adds them, it's just that sometimes the answer is off by 
one or two...")

Doctests are examples that you put into the documentation (usually into 
docstrings of functions and methods). Their primary purpose is to be 
examples for the reader to read, but the secondary aspect is that you 
can run the examples and be sure that they actually work as they are 
supposed to work.

Regression tests are to prevent bugs from re-occurring. For example, 
suppose you have a function spam() and you discover it fails for one 
particular input, "aardvark". The first thing you should do, before even 
fixing the bug, is write a test:

assert spam("aardvark") == "expected result"

This test will fail, because there's a bug in spam(). Now go ahead and 
fix the bug, and the test will then pass. If your code ever has a 
*regression* that returns the bug in spam(), the test will fail again 
and you will immediately notice. Regression tests are to prevent fixed 
bugs from returning.

User Acceptance Tests are mostly relevant when you're building software 
for a client. Both parties need a way to know when the software is 
"done". Of course software is never done, but if you're charging $10000 
for a software project, you need to have way of saying "This is where we 
stop, if you want us to continue, it will cost you more". UATs give both 
parties an objective way of telling that the promised functionality is 
there (e.g. "if the user clicks the Stop button, processing must stop 
within one second") and identifying bugs and/or areas that weren't 
specified in enough detail.

Of course, all of these are fuzzy categories -- there's no hard line 
between them.

> Test Driven Development tends to focus on the more normal inputs
> in my experience, but done properly your test code will usually
> be bigger than your production code. In a recent project that we
> completed we had 600k lines of production code and over a
> million lines of test code.

I can well believe that. I've had a look at a couple of my projects 
(*much* smaller than 600 KLOC!) and I'm averaging about 800 lines in the 
test suite per 1000 lines in the project. That's an over-estimate: the 
1000 lines includes doc tests, so should be counted towards the tests. 
Taking that into account, I get (very roughly) 1:1 ratio of test code to 
production code.


> And we still wound up with over 50 reported bugs during Beta test...
> But that was much better than the 2000 bugs on an earlier project :-)
> But testing is hard.

Maybe so, but nothing beats running your test suite and seeing 
everything pass!



-- 
Steven