coverage.py: "Statement coverage is the weakest measure of code coverage"

Ben Finney ben at benfinney.id.au
Sun Oct 28 18:56:48 EDT 2007


Howdy all,

Ned Batchelder has been maintaining the nice simple tool 'coverage.py'
<URL:http://nedbatchelder.com/code/modules/coverage.html> for
measuring unit test coverage.

On the same site, Ned includes documentation
<URL:http://nedbatchelder.com/code/modules/rees-coverage.html> by the
previous author, Gareth Rees, who says in the "Limitations" section:

    Statement coverage is the weakest measure of code coverage. It
    can't tell you when an if statement is missing an else clause
    ("branch coverage"); when a condition is only tested in one
    direction ("condition coverage"); when a loop is always taken and
    never skipped ("loop coverage"); and so on. See [Kaner 2000-10-17]
    <URL:http://www.kaner.com/pnsqc.html> for a summary of test
    coverage measures.

So, measuring "coverage of executed statements" reports complete
coverage incorrectly for an inline branch like 'foo if bar else baz',
or a 'while' statement, or a 'lambda' statement. The coverage is
reported complete if these statements are executed at all, but no
check is done for the 'else' clause, or the "no iterations" case, or
the actual code inside the lambda expression.

What approach could we take to improve 'coverage.py' such that it
*can* instrument and report on all branches within the written code
module, including those hidden inside multi-part statements?

-- 
 \            "Technology is neither good nor bad; nor is it neutral." |
  `\                       —Melvin Kranzberg's First Law of Technology |
_o__)                                                                  |
Ben Finney



More information about the Python-list mailing list