[Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Terry Reedy]

On 4/17/2011 1:32 AM, R. David Murray wrote:

> As Brett said, people do come to depend on the details of the
> implementation.  But IMO the PEP should be clarified to say that the
> tests we are talking about should be tests *of the published API*.
> That is, blackbox tests, not whitebox tests.

I think 100% *branch* coverage is barking up the wrong tree.
Better to say comprehensive *api* coverage. Bugs on the tracker 
generally come from not having that. (I am not saying 'all' to allow for 
bugs that happen from weird interactions or corner cases in spite of 
what could reasonably be called comprehensive._

> I don't think the PEP is asking this either (or if it is I agree it
> shouldn't be).  The way to get full branch coverage (and yes Exarkun is
> right, this is about individual branches; see coverage.py --branch) is
> to provide test cases that exercise the published API such that those
> branches are taken.  If you can't do that, then what is that branch
> of the Python code for?  If you can do that, how is the test case
> testing an implementation detail?  It is testing the behavior of the
> API.

Right.

-- 
Terry Jan Reedy