coverage.py: "Statement coverage is the weakest measure of code coverage"

[Ned Batchelder]

I don't know how to extend coverage.py to do more extensive checking,
but I know it would be both difficult and fascinating.  To help spur
some thought, I've sketched out some problems with statement coverage:
http://nedbatchelder.com/blog/20071030T084100.html

--Ned.

On Oct 28, 6:56 pm, Ben Finney <b... at benfinney.id.au> wrote:
> Howdy all,
>
> Ned Batchelder has been maintaining the nice simple tool 'coverage.py'
> <URL:http://nedbatchelder.com/code/modules/coverage.html> for
> measuring unit test coverage.
>
> On the same site, Ned includes documentation
> <URL:http://nedbatchelder.com/code/modules/rees-coverage.html> by the
> previous author, Gareth Rees, who says in the "Limitations" section:
>
>     Statement coverage is the weakest measure of code coverage. It
>     can't tell you when an if statement is missing an else clause
>     ("branch coverage"); when a condition is only tested in one
>     direction ("condition coverage"); when a loop is always taken and
>     never skipped ("loop coverage"); and so on. See [Kaner 2000-10-17]
>     <URL:http://www.kaner.com/pnsqc.html> for a summary of test
>     coverage measures.
>
> So, measuring "coverage of executed statements" reports complete
> coverage incorrectly for an inline branch like 'foo if bar else baz',
> or a 'while' statement, or a 'lambda' statement. The coverage is
> reported complete if these statements are executed at all, but no
> check is done for the 'else' clause, or the "no iterations" case, or
> the actual code inside the lambda expression.
>
> What approach could we take to improve 'coverage.py' such that it
> *can* instrument and report on all branches within the written code
> module, including those hidden inside multi-part statements?
>
> --
>  \            "Technology is neither good nor bad; nor is it neutral." |
>   `\                       -Melvin Kranzberg's First Law of Technology |
> _o__)                                                                  |
> Ben Finney