[py-dev] failing unit tests

[Martijn Faassen]

Hey Holger,

holger krekel wrote:
> [Martijn Faassen Fri, Nov 26, 2004 at 09:38:27PM +0100]
> 
>> Hi there,
>> 
>> I get a number of failing tests when I check out Py and run py.test
>> on it. If I read the output right, the failing tests are:
[snip]

> Yes, Ian Bicking checked these tests in to point to a certain import 
> problem.

Ah, I do recall seeing some discussion on the list, but I hadn't paid a
lot of attention.

> I just renamed it to "bugtest_import.py" so it doesn't get 
> automatically collected any more.

Great, I just saw that when I did an "svn up".

> Of course, you can still run the test by directly specifying the file
> with py.test FILENAME.

Right. This is now a bit more clear in the documentation. :)

>> Note that the test.py output is rather intimidating, especially if
>> you do not know the codebase you're testing.
> 
> What exactly do you find intimidating?

There was a lot of it in case of py, showing unfamiliar code, and then
there was the collected output. Just the quantity of unfamiliar
complicated looking things is intimidating. :)

This is of course usually fine and desirable if you're debugging a
familiar codebase, but below I tried to describe usage scenarios where
you want to avoid such intimidation.

> Note that there is a "failure" demo file in
> src/example/test/failure_demo.py which if run by py.test gives
> failure output examples for some 40 tests.

Right, I went through the documentation and ran into that failure demo.
Actually the failure demo output with some extra text would make for
good documentation by itself, I suspect.

>> This is a realistic scenario for any newbie to the codebase like
>> myself in this case, or possibly for deployment scenarios where
>> someone is asked to run the test suite just to answer the question
>> "which tests are failing?".
> 
> Yeah, well, probably at least partly because it relates to imports
> :-) More seriously, though, i tried to produce good output to show 
> what test is failing in which line (including an according call
> trace). It shows function context so that you don't neccessarily need
> to read the test file in your editor to understand the failure.

Yes, and for a developer this is fine. For a developer less familiar
with the codebase, or someone who is running the tests on other
purposes, a more minimalistic output option would be nice on occasion.

Of course part of the problem is that I modified some documentation text
and just to make sure wanted to run the test suite before checking it
in, even though to my knowledge py.test doesn't do doctets. I was
presented with rather scary detailed test failures, something I did not
expect at that point. I was expecting "everything is okay". :)

>> Perhaps a 'non-verbose' mode would be nice, or a 'sysadmin mode'
>> where only a bit of output is seen directly by the sysadmin, and
>> the rest is dumped into a file that can be emailed to the
>> developers.
> 
> What would you think about a dense summary at the end with one line 
> for each failing tests aka "filename:lineno XXXError" or so?

I think that would be useful. And then an option to suppress any output
but the summary and the nice dots before it. :)

>> Anyay, I also seem to have a lot of recorded stdout like this:
> 
> Yip, the test produces this via print statements.  For successful 
> tests the output is not shown, though. Do you find this questionable?
>  I have used it with nice results because you only get to see
> isolated print output for the failing tests which allows to use print
> statements even in central places and still you get meaningful
> test-related stdoutput.
> (Otherwise you get swamped by all the print output and don't know
> which output exactly related to the failure you want to debug).

I do not find this a questionable idea at all. I thought it was an 
intriguing idea when I read it in the documentation. I've edited the 
documentation somewhat to make the idea more clear to beginners. One 
thing I realized after editing the documentation is that I'm free to use 
'print' in whatever test code I like now, and even check it in, without 
causing this to outputted except when I really need it -- in case of 
test failure. Of course this still doesn't leave me free to do the same 
with application code. :)

Is that correct? If so, I shall edit the documentation accordingly.

I hope you like the edits I made to doc/test.txt.

Regards,

Martijn