Jeremy Hylton : weblog : 2003-10-14

Running Unit Tests

Tuesday, October 14, 2003, 7:42 p.m.

Python programmers need better tools for writing, running, and debugging unit tests. Ian Bicking's rant about unittest makes me think that Zope's test.py driver ought to find more widespread use.

Zope software tends to have a lot of unit tests. Zope3 has 5,000 tests and ZODB has about 1,500. Many of these test aren't, strictly speaking, unit tests. ZODB has tests that spawn a server process and spend a few minutes committing transactions against it to try to provoke race conditions. Nonetheless, they are all written using the unittest module and the test.py driver.

Bicking notes that the problem is really the default unittest driver:

Probably the issue here is that unittest.main is stupid. You can't add options (e.g., your own verbosity hooks), you can't easily indicate what tests you want to do... it's just dumb and not extensible.

Zope's test.py driver has a ton of features. We often had to struggle with unittest to implement them, but they are there. The short list is:

Jim Fulton has done a nice job of integrating doctest with unittest, so that you can write doctest tests and have them integrated into a unittest framework.

Any test driver has got to make some assumptions about how to find the tests to run and the code being tested. test.py has some fairly rigid rules that might not make sense for every project. All tests are integrated with the rest of the code in sub-packages. The test driver loads tests by importing them. If you've got a package foo, then test.py looks for tests in foo.tests. It searches for files in a tests package that begin with "test" and calls the test_suite() function it finds in them.

It is integrated with distutils for testing C extensions. It knows how to find code in the distutils build directory so that you can use setup.py to build an extension and test.py to test it.

The driver doesn't say anything about how you write the individual test suites, and Bicking has some good and funny complaints here. For example, "I do need setup and teardown (but not setUp and tearDown -- I'm sorry, but this is English, "setup" and "teardown" are two words, not four)."

His specific example is this TestCase class:

class DecodeTest(unittest.TestCase):
    def __init__(self, input, output):
        self.input = input
        self.output = output
    def test(self):
        self.assertEqual(decode(input), output)

This doesn't seem like the right way to start. Maybe I just don't see what it's supposed to do. If you create a lot of TestCase instances, then you don't have names for them and you can't get a simple summary of which tests passed and which failed.