[AstroPy] Testing Guidelines

[Mark Sienkiewicz]

Erik Tollerud wrote:
> In principle, the similarities between nose and py.test mean we
> wouldn't really have to make a decision until we need features that
> differ between them- developers could run whichever they prefer...
>   

If the decision goes that way, I can write up a document describing a 
common subset that covers most of what you need to do in routine 
testing.  Then you make your standard "use the astrolib-defined 
nose/py.test subset whenever it is practical".  Only a few unusual cases 
will need to deviate.


> Except there's one problem: the configuration files are formatted
> differently...  We want to use configuration settings for just one of
> the packages so that the suite always runs the same.

What configuration file is that?  I don't recall ever writing a 
configuration file for either nose or py.test.


>> By the way, it looks like there's been a lot of work on unittest in
>> python 2.7 (aka unittest2) so how does that compare to nose and
>> py.test?
>>     
>
> The biggest change I see is that it seems to now include a test
> discovery tool, but it is much less documented (a few paragraphs
> rather than many pages each for the other two), and seems to lack most
> of the nose and py.test runners' customization (like configuring in
> the setup.cfg file, easy plugins, etc.). It's an improvement, but it's
> still based around the more complex syntax of unittest, so most of
> what I said above as far as I can tell is still true.
>   


IMHO, the most interesting feature to me are the new comparison functions.

I never really liked all the weird "fail unless not less than or equal" 
kind of functions, but the nice thing in unittest2 is that some of the 
comparison functions are able to display useful reports about what the 
nature of the failure.  For example, I seem to remember a string compare 
that will split multi-line strings and display a diff-like result.

It strikes me as poor modularization, though:  there should be a 
separate library of comparison functions that can be used with any test 
framework, not a slightly different implementation in every test framework.

I think you can write a test function for nose or py.test and still use 
a unittest2 comparison function; you just have to be a little tricky.  
It's the kind of thing you would hide in a library.  Maybe we provide 
our own library of test helper functions.

The biggest negative of unittest2 is that it is still unittest.  When I 
first read about unittest, I thought "this is crazy complicated for 
running a simple test; I'm not going to use it".  I have a better 
understanding of it now, but I  still only ever use it when I have no 
other choice, and that basically means when I am working on code that 
somebody else already wrote using unittest.


> Regardless, we couldn't use that anyway if we're planning to stay
> 2.6-compatible.
>   

If you really like the new unittest, you can install unittest2 in python 
2.6.  The drawback is that each test file has to detect the three 
possible cases: new unittest, unittest2, or old unittest.

Mark S.