Code correctness, and testing strategies

[Ben Finney]

David <wizzardx at gmail.com> writes:

> On Sun, Jun 8, 2008 at 12:28 PM, Ben Finney
> <bignose+hates-spam at benfinney.id.au> wrote:
> 
> I did do it in a mostly top-down way, but didn't stop the BDD
> process to actually run the app :-)
> 
> It sounds like what you are suggesting is something like this:
> 
> 1) Following BDD, get a skeleton app working

Specifically, BDD requires that the behaviour be asserted by automated
tests. So, this skeleton should have automated tests for the behaviour
currently implemented.

> Then, your BDD process gets a few extra steps:
> 
> Old steps:
> 
> 1) Write a test which fails for [new feature]
> 2) Write code for [new feature] to pass the test
> 3) Refactor if needed
> 
> New steps:
> 
> 4) Run the app like an end-user, and see that it works for the [new feature]
> 5) Write an automated test which does (4), and verifies the [new
> feature] is working correctly

Rather, the test for the new feature should be written first, just
like any other test-implement-refactor cycle.

> Does this mean that you leave out the formal 'integration' and
> 'systems' testing steps? By actually running the app you are doing
> those things more or less.

I'm not sure why you think one would leave out those steps. The
integration and systems tests should be automated, and part of some
test suite that is run automatically on some trigger (such as every
commit to the integration branch of the version control system).

> Could you also leave out the unit tests, and just write automated
> acceptance tests?

They serve very different, complementary purposes, so you should be
writing both (and automating them in a test suite that runs without
manual intervention).

> Also, if you had to implement a few "user stories" to get your app
> into a skeleton state, do you need to go back and write all the
> missing acceptance tests?

Treat them as new behaviour. Find a way to express each one as a set
of "given this environment, and subject to this event, such-and-such
happens", where "such-and-such" is subject to a true/false assertion
by the test code.

> I have a few problems understanding how to write automated
> acceptance tests.

This is just another way of saying that determining requirements in
sufficient detail to implement them is hard.

True, and inescapable; but with a BDD approach, you only put in that
hard work for the behaviour you're *actually* implementing, avoiding
it for many false starts and dead-ends that get pruned during the
development process.

You also ensure the requirements are specified in at least one way: in
automated tests that are run as a suite by your build tools. Not
exactly perfect documentation, but it has the significant properties
that it will *exist*, and that divergence of the system from the
specification will be noticed very quickly.

> Perhaps you can reply with a few URLs where I can read more about
> this :-)

This weblog posting discusses unit tests and acceptance tests in agile
development
<URL:http://blog.objectmentor.com/articles/2007/10/17/tdd-with-acceptance-tests-and-unit-tests>.
I disagree with the author that there are *only* two types of tests
(as the comments discuss, there are other valuable types of automated
tests), but it does address important issues.

> There is no 'build' process (yet), since the app is 100% Python.

The "build" process does whatever is needed to prove that the
application is ready to run. For applications written in compiled
languages, it's usually enough to compile and link the stand-alone
executable.

For a Python application, "the build process" usually means preparing
it to be installed into the system, or some testing environment that
simulates such an environment.

A Python distutils 'setup.py' is a very common way to set up the build
parameters for an application.

> But I will be making a Debian installer a bit later.

That's an important part of the build process, but you should write
(and test via your automated build process) a 'setup.py' before doing
that.

> If I adopt BDD, my updated plan was to use it during app development
> and maintenance, but not for later testing. Do you suggest that
> after building a .deb in the chroot, the app should also be
> automatically installed under a chroot & acceptance tests run on my
> dev machine? Or should I package the acceptance tests along with the
> app, so that they can be (manually) run on test servers before going
> into production? Or do both?

BDD doesn't change in such a situation. The goal is to ensure that
desired behaviour of some part of the system is specified by automated
tests, that are all run as a suite triggered automatically by
significant changes to the system.

That description applies at any level of "part of the system", from
some small unit of code in a function to entire programs or larger
subsystems. The only things that change are which tools one uses to
implement and automated the tests.

> I've considered setting up a centralised build server at work, but
> currently I'm the only dev which actually builds & packages
> software, so it wouldn't be very useful.

It's extremely useful to ensure that the automated application build
infrastructure is in place early, so that it is easy to set up and
automate. This ensures that it actually gets done, rather than put in
the "too hard basket".

The benefits are many even for a single-developer project: you have
confidence that the application is simple to deploy somewhere other
than your development workstation, you get early notification when
this is not the case, you are encouraged from the start to keep
external dependencies low, your overall project design benefits
because you are thinking of the needs of a deployer of your
application, and so on.

While the project is small and the infrastructure simple is exactly
the right time to set up automated builds. When the time comes to make
the system and process more complicated, you'll have far more things
to do, and it's likely that you won't take the time at that point to
implement automated build systems that should have been in place from
the beginning.

> We do have other devs (PHP mostly), but they don't even use version
> control :-/.

Fix that problem first. Seriously.

-- 
 \       "The Stones, I love the Stones; I can't believe they're still |
  `\      doing it after all these years. I watch them whenever I can: |
_o__)                             Fred, Barney, ..."  -- Steven Wright |
Ben Finney