[Python-3000] doctests vs. unittests (was Re: pimp; restructuring the standard library)

[Mark Hammond]

Barry writes:

> On Jun 28, 2007, at 4:04 PM, Chris McDonough wrote:
>
> > I've historically not been a huge fan of doctests because (these
> > things may have changed since last I used doctest in anger):
>
> I used to think the same thing, but I've gotten the doctest
> religion.  I'm using them almost exclusively in the new
> Mailman code,
> and we use them at work (though both still have traditional Python
> unit tests).
>
> The thing that convinced me was the realization (assisted by my
> colleagues) that doctests are first and foremost
> documentation.  They
> are testable documentation sure, but the unit tests are secondary.

I admit I'm still yet to get the doctest religion to that degree - but for
exactly the same reasons :)

My problem is that too quickly, doctests go way beyond documentation - they
turn into a full-blown test framework, and this tends to work against the
clarity of the resulting documentation.

I like doctests that give you a general introduction to what is being
tested.  They can operate as a kind of 'tutorial', allowing someone with no
experience in the code to quickly see the basics of how it is used - that is
very useful indeed.

But IMO, these too quickly morph into the territory of unittests - they
start testing all corner cases.  The simple tutorial quality gets lost as
the doctests start including reams of test data and testing against
invariants that are important to the developer of the library, but mere
noise to a casual user of it.

Another key feature of unitests are their utility in helping you *find* bugs
in the first place.  When a bug is identified "in the field", unit tests
make it easy to find a "smallest possible" reproduction of a bug in order to
identify the root cause - which is then checked in when the bug is fixed.
If only doctests are available, then either that obscure bug is also added
to the doctests (making even more noise), or a test case is extracted to a
temporary program and discarded once the bug is fixed.

> A good test suite can benefit from both doctests and unittests and I
> don't think unittest will ever go away (nor should it), but in my
> latest work I'm opting more and more for doctests.

I find myself opting for doctests when working with "new" code, but quickly
leaving the doctests in their pristine state and moving to unittests once
the bugs get a bit curlier, or coverage.py directs me to write tests I'd
never dreamt of, etc...

> That Tim Peters is a smart guy after all I guess. :)

Indeed he is - which is exactly why use them as I described - that is my
interpretation of what he intended <wink>

Mark