[Python-Dev] We now have C code coverage!

[Nathaniel Smith]

On Fri, Jun 22, 2018 at 6:16 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 6/22/2018 8:43 PM, Terry Reedy wrote:
>>
>> On 6/22/2018 6:21 PM, Brett Cannon wrote:
>>>
>>> Thanks to a PR from Ammar Askar we now run Python under lcov as part of
>>> the code coverage build. And thanks to codecov.io <http://codecov.io>
>>> automatically merging code coverage reports we get a complete report of our
>>> coverage (the first results of which can now be seen at
>>> https://codecov.io/gh/python/cpython).
>>>
>>> And funny enough the coverage average changed less than 1%. :)
>>
>>
>> Questions:
>>
>> 1. Is it possible, given that we are not paying for those reports, to
>> customize the .coveragerc exclude_lines definitions?  Without such, the
>> idlelib measures are biased downward.
>>
>> 2. What do the colors of test files mean?  Every line of nearly all the
>> idlelib test files are executed, but over half are red.
>>
>> The Learn More page does not say anything about either.
>
>
> I discovered the answer to 2. by shift-clicking on a text_x file to see
> their coverage report for the file.  The colors actually do reflect the test
> lines executed.  codecov.io excludes gui tests*, so the reported coverage
> for tkinter, idlelib, and turtle is deceptive and bogus, and under-reports
> the total cpython coverage by a percent or two.  It would be better to
> exclude these modules.
>
> * I assume that codecov.io uses linux servers.  I have read that there are
> programs that simulate X-Windows so that gui code will execute without
> actual terminals.

Codecov.io doesn't run any tests itself; it's just a service for
aggregation and reporting. The coverage information is being gathered
while running CPython's regular CI tests, and then uploaded to
codecov.io to view.

So if you want to run the gui tests -- which seems like a good idea if
possible! -- then the way to do that would be to make them run as part
of the regular Travis/Appveyor/VSTS checks.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org