Hardware take on software testing.

[Tom Verbeure]

> We can find hard to get at bugs with this technique in HW design,
> that's why I'd also like it added to the software arsenal.

There are some major differences between hardware and software develoment
that make directed random testing (DRT) much less efficient for software.

Conceptually, a software program is basically one HUGE statemachine where
every variable in the data memory influences the progress of the state
machine (even if you don't usually think like it that way.)

In hardware, you also have a whole bunch of explicitly designed state
machines that work in parallel. Those statemachines will change state based
on the input simuli and the time at which they arrive. The goals of directed
random testing in hardware is to find cases where one or more statemachines
get tricked into certain corner cases that are hard to trigger manually. 2
parallel but interacting statemachines come to mind here. By randomly
changing input data AND the time at which it arrives for a long time, you
can be reasonably sure that you trick the hardware into most corner cases.

This technique is not very useful for software, especially if it's single
threaded: if the software isn't interactive, there isn't much chance of
problems due to arbitration between parallel events, since there aren't any.
:-)  (A lot of DRTs are written to find arbitration errors between
statemachines.) I

In addition, to make DRT really useful, you need a golden model to verify
that the results generated by randomly generated data is correct. This means
you need to write a certain piece of software twice, with exactly the same
higher level behaviour.
How would you verify, say, a Python interpreter with random testing? By
writing one in C and another one in Java, throwing random instructions at
them and checking the behaviour, right? Well, you just created 2 highly
complex multi-man-year projects. :-)

Due to the huge up front costs and turn around time, it makes sense for
hardware to make a golden model. Not in the least because it's very easy to
describe complicated hardware with simple logical statements.
Eg. a linked list manager in hardware can cost you weeks of design time, but
in your verification language (eg. Specman's 'e', it's 1 line of code.) That
being the case, the cost of writing a gold model is minimal compare to the
cost to design the hardware. For software, it's the same effort... twice.

In other words, for software, it's not cost effective, since it's usually
quite easy to make bugfixes in later revisions and tell a customer not to
use a particular feature for a while or quickly ship a patch. It's much
harder to tell the user of an IP router chip to avoid packets that contain a
certain sequence of bytes and to wait 6 months for a new revision of the
chip that can then be replaced on the PCB.

> I don't doubt that XP and TDD can be an advance on previous methods,
> I'm just wary of the lack of metrics. And I'm unsure of how you
> alleviate the problem of only testing how one person or close team
> think it should work. In HW design some teams have separate
> Verification and Design groups and/or create two implimentations in
> different languages and compare their results. If you'r writing the
> test and writing the code, you can be blind to common errors.
> Is TDD enough?

Apparently it is good enough for most people, since virtually nobody will go
farther than that.

Tom