Hardware take on software testing.

[Paddy McCarthy]

If software were tested like hardware...

* You would use constrained random generation [1] of data and collect
program statement coverage [2] metrics, as well as branch coverage [3]
metrics and state machine coverage [4] metrics.

* You would generate another implimentation of some aspects of parts
of your program, as well as your whole program and either formally
prove that the two programs are equivalent within the domain of the
second implimentation, or be able to check during the run of extensive
test cases, that the two implimwentations are equivalent.

Mind you, if Hardware Description Languages were more like python...

* HDLs would be developed in an interpretive environment.
* HDL agregates, such as modules/structs/functions would have
docstrings.

I ask because I value the view of Pythonistas and if you look at the
electronic/Software products you buy today, a lot go out the door with
bugs. The software industry has relied on the fact that it doesn't
cost them much to make another release but if you look at the
proliferation of updates out there it doesn't seem to be working.
Customers should expect more.

Cheers, Paddy.


Key.
[1] Constrained random generation: used in the test harness, Think of
being able to state that a referenced variable will return a random
value from a set (usually a set of integers). The members of the set
of valid values can in turn depend on the values of other variables
which may change during the test, and may be randomised themselves.
Usually a constraints solver is used behind the scenes to
automatically resolve multiple constraints on a variable.
Example; a=Rand(range(8)); b.c =
Rand(range(256));d.e.f[3]=Rand(range(41))
Constrain b.c <= (a+1)*d.e.f[3]
after resolving the constraints, each reference to the value of b.c
would return a value within its initial range and satisfying all the
constraints accessing the value of a, to return a new a value would
then then automatically change the set of values returned randomly
from b.c as they are linked by the constraint.
This is used to generate a large amount of valid test data that can be
'steered' by adding and altering constraints.

[2] Statement coverage: SSSSSoftware tools do this now. You can find
out how many times each statement is executed during a test run.

[3] Branch coverage: gives information on conditional statements. For
example if you had the if statement:
if (s==t) or (u==v) or w: ...
Statement coverage would count how many times each sub-expression
caused the statement to be true, ie counts for s--t, u==v, and w.

[4] State machine coverage: Collects statistics on the transitions,
and sets of transitions taken in state machines you define in your
program.