doctest.py and testing frameworks (was Re: [Python-Dev] Fire, may I present accellerant. Accellerant, thi...)

[Peter Funk]

Hi,

Mitchell Morris wrote to me:
> Peter:
> 
> Thanks for reading my post, and I especially appreciate your taking 
> the time and effort to respond. I also appreciate that you didn't take 
> me to task for using "it's" instead of "its" as the possessive form. 
> (I've been completely reeducated now, I assure you). As I'm not 
> currently a subscriber to python-dev, I don't know if this should 
> have been replied to the list or directly to you, so I'm sending it to 
> you only. If you feel it appropriate, please redistribute it as you 
> wish.

[...I wrote to Mitchell:...]
> > I've used doctest in my own code and found the idea to integrate
> > test cases as sequences of calling code and expected result tuples
> > as documentation (exmaples of use) into the doc string of modules, 
> > classes and methods very appealing.
> > 
> > However this promising approach needs further work:  Especially
> > GUI parts can't be tested this way and in larger contexts, it 
> > is indeed very tedious to place *all* test code into doc strings.  
> > Often you need to prepare a defined starting point for the tests
> > of several methods.  It is not easy to decide, where to put this.
> 
> I used doctest rather extensively a little while ago, but it was only 
> the vanilla variant originally posted by Tim. I thought it was really an 
> excellent work, but ran into similar stumbling blocks. The largest 
> issue for me was the aggregation of several modules worth of tests 
> into larger chunks, but I now know (from what I had to build to 
> complete the test plan I was working with) that I would very shortly 
> have run into the problems of replicating the test environment: 
> preparing input files/directories/databases, creating all the 
> supplementary bits and pieces, etc ad nauseam.
> 
> A secondary issue which none of these tools address yet, but I feel 
> will be significant eventually, is the explicit naming of inter-test 
> dependence. If my class B tests require using some functionality 
> from class A, I would want the A tests to succeed before trying the 
> B tests at all (assuming I'm going to run both A and B in a single 
> iteration). I suspect, then, that one of these candidates will require 
> additional work to be our complete solution; I want to make sure 
> that I/we are working on the best candidate possible before 
> spending the effort.
> 
> 
> > > 	again, additional effort could obviously provide the missing
> > > 	functionality. On the other hand, doctest.py isn't currently
> > > 	part of the standard distribution.
> > 
> > Yes. :-(  
> 
> This, I feel, represents a significant problem. We should have a 
> testing framework as part of the core distribution. 

I agree.

> We have HTML 
> tokenizing, mail sending, RFC822 parsing, /bin/sh-like lexing, etc 
> etc but don't feel that providing test support to our community of 
> developer is appropriate? Bizarre. Perhaps even more bizarre is 
> that there is a non-trivial regrtest.py that is part of the distribution, 
> but it's hidden away and not documented. 

Perhaps because no one volunteered to write documentation for it.
Writing docs is an tedious task especially if you are no native
english speaker.

> I think we are not 
> providing all the service to the Python community that we could 
> and should.

You are welcome to do something about this.

> One of the strengths of Python, in my experience, has been the 
> ease with which code is written. This makes it much easier to 
> convince yourself and others to refactor mercilessly; the kinds of 
> cruft that grow on C++ solutions are rarely seen in Python code 
> because we can repartition and rewrite our work much more easily. 
> Providing a standard means to verify functionality before refactoring 
> makes it much safer to proceed, since we can "prove" (for a very 
> loose definition of "prove") that we haven't broken anything that 
> anybody else cares about. I think this is a direction we should 
> encourage, both through example and explicit 
> direction/documentation.
> 
> 
> > I attach my version of 'doctest' to this email.  Do what you want with it.  
> > Currently I favour 'doctest' over 'pyunit'.
> 
> If you don't mind, could you elaborate why you prefer doctest, or at 
> least what you find unpleasant about pyunit? I would like to include 
> this in the document for future reference, and you may even 
> convince me to change my vote for doctest.py instead.

I must admit that I didn't had a look at PyUnit since January.  

I've just downloaded pyunit-1.2.0 and had a closer look at it.
Nevertheless it is very hard to beat the elegance of Tim Peters
approach to cut'n'paste test cases from the interpreter prompt.
Although you usually get 2 lines for each test case, building the
test cases is much easier in the first place.

May be both approaches to unit testing can be integrated somehow?
Steal the concept of 'fixtures' from PyUnit and steal the idea to
define simple test cases in the doc strings from 'doctest'.  It shouldn't
be that hard to add a semiautomatic import hierarchy traversal tool.

Regards, Peter