When is unit-testing bad? [was: Re: does lack of type...]

[David Bolen]

"Aurélien Géron" <ageron at HOHOHOHOvideotron.ca> writes:

> There are things that are hard to unit test (I'm talking about automated
> unit tests here) and which you just can't do without in some projects.  For
> instance, the database!  I worked in a couple of projects with automatic
> unit tests for the DB related components, and having lived through it, I'm
> not sure that it was worth it at all!  Manual unit tests would have made
> much more sense.  (...)
> (...)
> I guess my point is that though you may be right in some specific contexts,
> I don't think you can generalize.  That's IMHO of course.

I'd agree that there are elements that are much more difficult than
others to test in an automated fashion - sometimes impractically so
given the cost to benefits comparison.  Personally, I'd agree that
databases, GUIs and asynchronous activities such as network or
multi-threading are areas that fall into that category.  But I also
think that such areas can often be tested far more than some might
think within a given development effort.

The idea is to strip any layer that can't be (or for a given project
can't be justified to be) automatically tested down to its bare
minimum, because an automated test will almost always be of more value
over time (with repeated use, different developers, etc...) than a
manual test.

This is where using the tests during development (as opposed to
afterwards) can help ensure a test friendly design, and one that is
often more modular and structured than a typical developer might have
otherwise arrived at without the tests during development.

For example, when writing a DB interface layer or component (whether
for persistence or direct RDBMS access), if you're trying to construct
tests as you develop, you will more than likely arrive at a structure
where the final database access layer is isolated enough that you can
mock it up for most of your tests of the remainder of your code - if
only because without it you find yourself not able to construct tests :-)

This might take the form of a specific component used to wrap raw
database access.  Or, if it's a SQL database, you might find yourself
designing business components such that they serve as producers of the
actual SQL, for which a separate object consumes it and executes it
against a database.  That interface (the SQL output) provides a test
point for the business logic without involving the database.

In the former case, tests would simulate the lowest level database
layer to function just as is necessary for the test (accepting and
returning appropriate data), thus being able to rapidly and
efficiently test the remainder of the code without a live database in
an automated and controlled fashion.  In the latter case, the tests
would execute the business logic and compare the result to expected
SQL.  A much smaller suite of tests to exercise the database layer
would use a test database (or equivalent) but would be focused
strictly on the raw database interface, or execution of known SQL test
statements, and should be much more containable (in time and
development effort), and probably still automatable, although perhaps
on a longer cycle time.

If the system wasn't designed with testing involved, the actual
database I/O might very well be embedded in a higher layer (such as
the persistence mechanism which might be tightly coupled to an object
layer) and hard to isolate from the rest of the logic.  For example,
you might find business objects directly calling a persistence or
database I/O layer within their implementation, making it hard to test
the business logic without a live database.

Similar ideas apply to GUIs - keep the actual interface layer as
extremely thin as possible, and containing no business logic or
application decision processing.  Whether you use MVC, MVP or an
equivalent approach, ensure that you can test all underlying business
logic without actually needing a GUI.  This holds for applications and
web sites (in many respects, a web site can just be modeled as a
different form of view).

For example, in MVC/MVP, your automated test can call
controller/presenter methods to simulate any aspect of user
interaction with the UI, but without the need for a UI, and can mock
up a non-UI view to ensure that the model is notified of appropriate
changes and that queries to it reflect the new information.

At some point you need to verify that performing action X against UI
object Y generates the right requests to the other system components,
but that's far more constrained testing then trying to test the
business logic through that UI.  And yes, that could probably involve
external UI testing tools, or some manual interaction with an
automated test mocking the underlying controller/presenter and model.

Of course, this is all easiest if you are designing a system with
testability in mind from the get go.  Trying to retrofit unit tests
onto a GUI for example, where much of the business logic is attached
to specific UI elements and not reachable without going through the UI
is just plain old difficult.  The idea of making up tests as you go
(whether TDD or equivalent) and using them as part of the development
process helps not only ensure that the final product is as tested, in
as automated a fashion, as possible, but that it ends up being
designed to be tested.

-- David