[py-dev] failing unit tests

[holger krekel]

Hi Martijn, 

[Martijn Faassen Fri, Nov 26, 2004 at 10:32:33PM +0100]
> holger krekel wrote:
> >[Martijn Faassen Fri, Nov 26, 2004 at 09:38:27PM +0100]
> ...
> Right, I went through the documentation and ran into that failure demo.
> Actually the failure demo output with some extra text would make for
> good documentation by itself, I suspect.

Yes, i agree. 
 
> >Yeah, well, probably at least partly because it relates to imports
> >:-) More seriously, though, i tried to produce good output to show 
> >what test is failing in which line (including an according call
> >trace). It shows function context so that you don't neccessarily need
> >to read the test file in your editor to understand the failure.
> 
> Yes, and for a developer this is fine. For a developer less familiar
> with the codebase, or someone who is running the tests on other
> purposes, a more minimalistic output option would be nice on occasion.

It seems reasonable to add an option.  Although this might not help a 
complete newbie as he wouldn't know about the option. 

> Of course part of the problem is that I modified some documentation text
> and just to make sure wanted to run the test suite before checking it
> in, even though to my knowledge py.test doesn't do doctets. I was
> presented with rather scary detailed test failures, something I did not
> expect at that point. I was expecting "everything is okay". :)

Yes, and rightfully so.  I should have thought about this earlier. 
Somehow i didn't think that somebody would get irritated by it in the 
last days and i also was too consumed by the pypy sprint :-) 
 
> >What would you think about a dense summary at the end with one line 
> >for each failing tests aka "filename:lineno XXXError" or so?
> 
> I think that would be useful. And then an option to suppress any output
> but the summary and the nice dots before it. :)

something like "--kiss" because it's a good occassion to mention 
this pythonic principle :-) Or maybe just "--terseoutput" or "-t" 
or something.  This option would be tied to the TextReporter though. 
For generating html-output (definitely a plan for the near future) 
the mileage will vary i suppose. 

> >Yip, the test produces this via print statements.  For successful 
> >tests the output is not shown, though.  Do you find this questionable?
> 
> I do not find this a questionable idea at all. I thought it was an 
> intriguing idea when I read it in the documentation. I've edited the 
> documentation somewhat to make the idea more clear to beginners. One 
> thing I realized after editing the documentation is that I'm free to use 
> 'print' in whatever test code I like now, and even check it in, without 
> causing this to outputted except when I really need it -- in case of 
> test failure. Of course this still doesn't leave me free to do the same 
> with application code. :)
> 
> Is that correct? If so, I shall edit the documentation accordingly.

That is correct.  There is a small glitch with this otherwise
nice feature, though:  I sometimes spread print statements
over all my code and work with the failing test-output.  Then
the tests pass and i think "cool, checkin time" but of
course i still have to get rid of the print statement in my app code. 
Currently py.test doesn't help with this task and i am not sure yet
how to do it (except from some fragile hacks i would envision to 
reside in the 'magic' part of the py lib :-). 
 
> I hope you like the edits I made to doc/test.txt.

They are great!  I saw you introduced some references to unittest.py 
which i usually avoided to not confuse people who don't know
anything about testing so far.  But by now i guess (and Ian said this
earlier already) it's probably better to mention it every now and then 
for developers coming from unittest.py. 

cheers, 
    
    holger