[Python-3000] doctests vs. unittests (was Re: pimp; restructuring the standard library)

[Ron Adam]

Mark Hammond wrote:
> Barry writes:
> 
>> On Jun 28, 2007, at 4:04 PM, Chris McDonough wrote:
>> A good test suite can benefit from both doctests and unittests and I
>> don't think unittest will ever go away (nor should it), but in my
>> latest work I'm opting more and more for doctests.
> 
> I find myself opting for doctests when working with "new" code, but quickly
> leaving the doctests in their pristine state and moving to unittests once
> the bugs get a bit curlier, or coverage.py directs me to write tests I'd
> never dreamt of, etc...

I agree with this completely.  Doctests are very useful for getting the 
basics down and working both while the code is being written.

After that, unittests are much better for testing edge cases and making 
sure everything works including the kitchen sink, the pipes to the sink, 
the quality of water, etc...  ;-)

IF there is a problem, I don't think it is in the exact execution of 
doctests or unittests, but in the organization of them relative to the 
modules and how they are run.


Currently the unittest test suite runs tests that are in a know place with 
known name.  There can be modules in a distribution that are completely 
untested and you would not know unless you manually checked for this.

I'd like to see this turned around a bit so that the test suite runner 
first scans the modules and then looks for tests for each of them.  And if 
no test for a particular module is found, give some sort of warning.

Possibly a module could have a __tests__ 'list' attribute with locations of 
tests?  So an automatic test runner might start by first importing a module 
and then running the test modules listed in __tests__.  And yes even the 
tests can have tests. ;-)  A "__tests__ = None" could explicitly turn that 
off, where a "__tests__ = []" would indicate a module that does not yet 
have tests but needs them.

This could also reduce the boiler plate needed to run unittests as a side 
bonus.


There's been a few times where I started writing doctests for a module with 
less than 100 lines of code and by the time I was done with the doc tests, 
it became a 500 line or more module.  The actual code then starts to get 
lost in the file.

It would be cool if the documents files could also contain the doc tests 
instead of them being in the source code.  I'm sure the could be done now, 
but there isn't a standard way to do this.  Currently I create a seperate 
test module which unclutters the program modules, but then it isn't clear 
these are meant to be documentation first.

Cheers,
    Ron