[Python-Dev] unit testing bake-off

[Jeremy Hylton]

There was a brief discussion of unit testing last millennium, which did
not reach any conclusions.  I'd like to restart the discussion and set
some specific goals.  The action item is a unit testing bake-off, held
next week, to choose a tool.

The primary goal is to choose a unit testing framework for the
regression test suite.  Tests written with this framework would
eventually replace the current regrtest.py framework, based on
comparing test output to expected output.

For the 2.1 release, the goal would be to choose a test framework to
include in the standard distribution and use it to write some or all
of the new tests.  We would need to integrate it in some way with
regrtest.py, so that a single command can be used to run all the
tests.

In the long run, we can migrate existing tests to use the new system.
The new system can help us address some other goals:

    - running an entire test suite to completion instead of stopping
      on the first failure

    - clearer reporting of what went wrong

    - better support for conditional tests, e.g. write a test for
      httplib that only runs if the network is up.  This is tied into
      better error reporting, since the current test suite could only
      report that httplib succeeded or failed.

Does anyone disagree with the goal?

Three tools have been proposed: PyUnit, Quixote unittest, and doctest.

doctest has been championed by Peter Funk, who wants a few new
features, but Tim, its author, isn't pushing it as a tool for writing
stand alone tests.  I think the best way to use doctest is for module
writers to consider it when writing a new module.  If doctest is used
from the start for a module, we could integrate it with the regression
test.  It seems quite useful for what it is intended for, but is not a
general solution.

That leaves PyUnit and Quixote's unittest.  The two tools are fairly
similar, but differ on a number of non-trivial details.  Quixote also
integrates code coverage, which is quite handy.  If we don't adopt its
unittest, we should add code coverage to PyUnit.

Is anyone else interested in the choice between the two?  If so, I
suggest you try writing some tests with each tool and reporting back
with your feedback.  I propose leaving one week for such a bake-off and
making a decision next Friday.

Jeremy