Debugging doctest failures

[Eric Mahurin]

Noob here.  Just got into python a little over a week ago...

One of the (unique?) things I really like about python is the concept
of doctesting.  But, now I want more!  Here's what I'd like to see:

* easy debugging.  As soon as there is a failure (unexpected exception
or mismatch), drop down into the debugger in a way to isolates the bug
down to the nearest test/example as much as possible.

* integrated with code coverage.  I'd like to be targetting 100% code
coverage with doctests.  With full (or a target) coverage, no
additional coverage data would be needed and otherwise something that
tells which lines don't have coverage would be good.  A way to mark
code that you don't care about covering (typically error conditions -
at least initially) would also be good.

* whether stdout and/or sterr should be checked.


If anybody has any pointers to what I should be doing with doctest or
another framework I should be using (zope.testing?), I'd appreciate
it.

I made a crack at the first one above (easy debugging).  Here is a
little script I came up with.  You just give modules you want to
doctest as arguments.  When a normal failure occurs, it will restart
the test (with verbose) and set traces before each example.  When an
unexpected exception occurs, it pulls up the debugger with the
backtrace and continuing will restart the test just like a normal
failure.  I've used this a bit and it allows me to find bugs very fast
without changing the code a bit for debug purposes.  The
implementation below is quite hacky and tied to the implementation
within the doctest module.


#!/usr/bin/env python

import doctest
import sys

def trace_runner(reporter,arg3) :
    import sys
    tb = sys.exc_info()[2]
    while tb.tb_next : tb = tb.tb_next
    l = tb.tb_frame.f_locals
    out, test, example, arg3 = l['out'], l['test'], l['example'],
l[arg3]
    runner = doctest.DocTestRunner(verbose=True,
        optionflags=doctest.NORMALIZE_WHITESPACE)
    getattr(runner,reporter)(out, test, example, arg3)
    for e in test.examples :
        e.source = 'pdb.set_trace();' + e.source
    exec 'import pdb' in test.globs
    return runner

mod_names = []
for mod_name in sys.argv[1:] :
    __import__(mod_name,globals(),locals())
    mod_names.append(mod_name)
finder = doctest.DocTestFinder()
runner = doctest.DebugRunner(optionflags=doctest.NORMALIZE_WHITESPACE)

try :

    for mod_name in mod_names :
        mod = sys.modules.get(mod_name)
        for test in finder.find(mod, mod_name) :
            runner.run(test)

except doctest.DocTestFailure :

    runner = trace_runner('report_failure','got')
    runner.run(test)

except doctest.UnexpectedException, err :

    runner = trace_runner('report_unexpected_exception','exc_info')
    import pdb
    pdb.post_mortem(err.exc_info[2])
    runner.run(test)