[Tutor] the art of testing

[Alan Gauld]

"Serdar Tumgoren" <zstumgoren at gmail.com> wrote 

> Simple enough in theory, but the problem I'm hitting is where to begin
> with tests on data that is ALL over the place.

I've just spent the last 2 days in a workshop with >30 of our company's 
end to end test team. These guys are professional testers, all they do 
is test software systems. Mostly they work on large scale systems 
comprising many millions of lines of code on multiple OS and physical 
servers/networks. It was very interesting working with them and learning 
more about the techniques they use. Some of these involve very esoteric
(and expensive!) tools. However much of it is applicable to smaller 
systems.

Two key principles they apply:

1) Define the "System Under Test" (SUT) and treat it as a black box.
This involves your test boundary and working out what all the inputs 
are and all the outputs. Then you create a matrix mapping every set 
of inputs (the input vector) to every corresponding set ouf outputs
(The output vector). The set of functions which maps the input to 
output is known as the transfor function  matrix and if you can define 
it mathematically it becomes possible to automate a compete test 
cycle. Unfortunately its virtually never definable in real terms so we 
come to point 2...

2) Use risk based testing
This means look at what is most likely to break and focus effort on those 
areas. Common breakage areas are poor data quality and faulty 
interfaces. So test the inputs and outputs thoroughly.

> In such a case, should I be writing test cases for *expected* inputs
> and then coding the the parser portion of my program to handle the
> myriad of possible "bad" data?
> 
> Or given the range of possible inputs, should I simply skip testing
> for valid data at the parser level, and instead worry about flagging
> (or otherwise handling) invalid input at the database-insertion layer
> (and therefore write tests at that layer)?

The typical way of testing inputs with ranges is to test
just below the lower boundary, just above the boundary, the mid point, 
just below the upper boundary, just above the boundary known invalid 
values, "wildy implausible" values.

Thus for an input that can accept values between 1 and 100 you 
would test 0,1,50,100,101, -50 and 'five' say

Its not exhaustive but it covers a range of valid and invalid data points.
You could also test very large data values such as
12165231862471893479073407147235801787789578917897
Which will check for buffer and overflow type problems

But the point is you applyintelligence to determine the most likely 
forms of data error and test those values, not every possible value.

> Or should I not be testing data values at all, but rather the results
> of actions performed on that data?

Data errors are a high risk area therefore should always be tested.
Look to automate the testing if at all possible and write a program 
to generate the data sets used and ideally to generate the expected 
output data too - but thats hard since presumably you need the SUT 
to do that!

> It seems like these questions must be a subset of the issues in the
> realm of testing. Can anyone recommend a resource on the types of
> tests that should be applied to the various tasks and stages of the
> development process?
> 
> A friend recommended The Art of Software Testing -- is that the type
> of book that covers these issues? 

Yes and is one of the stabndard texts.
But most general software engineering texts cover testing issues.
For example try Somerville, Pressman, McConell etc.

> suitable alternative that costs less than $100?

Most of the mentioned authors have at least 1 chapter on testing.

HTH,

PS. It was also interesting to hear how testing has moved on from 
the days I spent as a tester at the beginning of my career in 
software engineering. In particular the challenges of Agile techniques 
for E2E testing and the move away from blanket testing to risk based 
testing, as well as the change in emphasis from "try to break it" - the 
dominant advice in my day - to "check it breaks cleanly under real loads"

-- 
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/