Notes on unittest.py

[Andrew Kuchling]

I've just sent off a note to c.l.py.announce about the Quixote 0.10
release.  One of the interesting bits hiding inside the tarball is a
nifty unit testing framework, unittest.py, and I'm writing this post
to encourage more people to pick it up and use the framework, since
there's nothing Quixote-specific about it.

(Much of the rest of this post is taken from the unittest.py Web page
which is part of the Quixote Web page at
http://www.mems-exchange.org/software/python/quixote/ .)

unittest.py was originally written by Greg Ward, and Neil Schemenauer
added the code coverage features using a modified version of Skip
Montanaro's trace.py.

Consider a simple Python function f() that takes a string s and
multiplies it by a value val, but reports an error if val is negative.

def f(s, val):
    if val < 0:
        raise ValueError, 'val cannot be negative'

    return s * val

The test suite for this function might be:

=========================
from quixote.test.unittest import TestScenario, parse_args, run_scenarios

import module

tested_modules = ['module']

class MyFunctionTest (TestScenario):

    def setup(self):
        pass

    def shutdown(self):
        pass

    def check_val_param(self):
        "Test error checking for the val parameter: 3"
        # Negative number should raise a ValueError
        self.test_exc( "module.f('', -1)", ValueError)
        
        # Zero is OK
        self.test_stmt( "module.f('', 0)")

        # Positive numbers are also OK
        self.test_stmt( "module.f('', 1)")

    def check_func(self):
        "Test the function's output: 6"

        # Test the null case (val == 0)
        self.test_val( "module.f('', 0)", '')
        self.test_val( "module.f('abc', 0)", '')

        # Test the identity (val == 1)
        self.test_val( "module.f('', 1)", '')
        self.test_val( "module.f('abc', 1)", 'abc')

        # Test a real case (val == 3)
        self.test_val( "module.f('', 3)", '')
        self.test_val( "module.f('abc', 3)", 'abcabcabc')

if __name__ == "__main__":
    (scenarios, options) = parse_args()
    run_scenarios (scenarios, options)
=========================

When run, this test case prints:
kronos /tmp>python test.py
MyFunctionTest:
  ok: Test the function's output ('func') (6 tests passed)
  ok: Test error checking for the val parameter ('val_param') (3 tests passed)
ok: 9 tests passed
kronos /tmp>

Other available methods for defining unit tests are:

     * test_stmt(stmt)
       Execute the statement stmt, assuming it will run without raising
       an exception.
     * test_exc(stmt, exception)
       Execute the statement stmt, assuming it will raise the exception
       exception.
     * test_val(code, value, match_ident=0, match_types=0)
       Test whether the expression code returns value. The optional
       match_ident and match_types flags allow enforcing object identity
       or type identity.
     * test_bool(code, want_true=1)
       Test whether the Boolean return value of the expression code is
       equivalent to the want_true flag.
     * test_seq(code, sequence, match_types=0, match_order=1)
       Test whether the expression code returns a sequence that matches
       sequence. If match_order is false, the order of the sequence is
       assumed to be irrelevant, so that only its contents matter.

The unit test framework also supports measuring code coverage, using a
modified version of Skip Montanaro's code coverage tool.  The default
argument parsing lets you add the -c switch to turn on code coverage,
and adding -v produces a listing of the tested module highlighting
lines that weren't executed.  For example:

kronos /tmp>python test.py -c
MyFunctionTest:
  ok: Test the function's output ('func') (6 tests passed)
  ok: Test error checking for the val parameter ('val_param') (3 tests passed)
ok: 9 tests passed
code coverage:
  module: 100.0% (4/4)
kronos /tmp>python test.py -c -v 
  ... additional output while running the tests deleted ...
ok: 9 tests passed
code coverage:
  module:
      .
     10: def f(s, val):
      9:     if val < 0:
      1:         raise ValueError, 'val cannot be negative'
      .
      8:     return s * val
  100.0% (4/4)

The number at left is the number of times each line was executed.

If you add an 'elif val == 42' branch to the 'if' statement, its 
block will never be executed by the tests, so the code coverage
reports the unexecuted lines:

code coverage:
  module:
      .
     10: def f(s, val):
      9:     if val < 0:
      1:         raise ValueError, 'val cannot be negative'
      8:     elif val == 42:
  >>>>>>         print 'The answer!'
      .
      8:     return s * val
  83.3% (5/6)
kronos /tmp>        

Armed with this information, you can now go back and add a test that
will exercise the val==42 branch.

A run_tests.py script is also included that can run a single test, or
will look for subdirectories named test/ and run all the tests in
them.  This is useful for testing an entire source tree:

kronos proto3>~/src/mems/tools/run_tests.py -r .
looking for test scripts...found 45
ok: lib/test/test_pvalue.py: 118 tests passed
ok: lib/test/test_range.py: 129 tests passed
ok: lib/test/test_unit.py: 109 tests passed
...
ok: template/test/test_eqtemplate.py: 14 tests passed
ok: prc/test/test_inter.py: 0 tests passed
ok: 2104 tests passed
kronos proto3>           

At work we run the full test suite every night from a cron job in
order to catch errors.

Personally, I've found using this testing framework to be a delight;
writing tests doesn't feel clunky, and failing test cases usually
report enough diagnostic information to fix the root problem.  It
would be great to see more people using the test suite in their own
projects, and maybe it can be considered for addition to the standard
library in Python 2.1, in competition with other frameworks such as
PyUnit (pyunit.sourceforge.net).

--amk