[Python-Dev] unit testing and Python regression test

[Tim Peters]

[Jeremy Hylton]
> There was recently some idle chatter in Guido's living room about
> using a unit testing framework (like PyUnit) for the Python regression
> test suite.  We're also writing tests for some DC projects, and need
> to decide what framework to use.
>
> Does anyone have opinions on test frameworks?  A quick web search
> turned up PyUnit (pyunit.sourceforge.net) and a script by Tres Seaver
> that allows implements xUnit-style unit tests.  Are there other tools
> we should consider?

My own doctest is loved by people other than just me <wink>, but is aimed at
ensuring that examples in docstrings work exactly as shown (which is why it
starts with "doc" instead of "test").

> Is anyone else interested in migrating the current test suite to a new
> framework?

Yes.

> I hope the new framework will allow us to improve the test
> suite in a number of ways:
>
>     - run an entire test suite to completion instead of stopping on
>       the first failure

doctest does that.

>     - clearer reporting of what went wrong

Ditto.

>     - better support for conditional tests, e.g. write a test for
>       httplib that only runs if the network is up.  This is tied into
>       better error reporting, since the current test suite could only
>       report that httplib succeeded or failed.

A doctest test is simply an interactive Python session pasted into a
docstring (or more than one session, and/or interspersed with prose).  If
you can write an example in the interactive shell, doctest will verify it
still works as advertised.  This allows for embedding unit tests into the
docs for each function, method and class.  Nothing about them "looks like"
an artificial test tacked on:  the examples in the docs *are* the test
cases.

I need to try the other frameworks.  I dare say doctest is ideal for
computational functions, where the intended input->output relationship can
be clearly explicated via examples.  It's useless for GUIs.  Usefulness
varies accordingly between those extremes (doctest is natural exactly to the
extent that a captured interactive session is helpful for documentation
purposes).

testing-ain't-easy-ly y'rs  - tim