[AstroPy] Testing Guidelines

Sat Jul 30 06:07:10 EDT 2011

(To summarize, if this message is TL;DR: After looking more closely,
I'm thinking py.test fits our purposes because it seems to do be able
to accept syntax from unittest, doctest, nose, and its own all at
once, while the others are more limited.  It also may be better for
future plugins, and it doesn't have be installed for the test suit to
be run by users, unlike nose.)

> Let's keep the discussion going on testing frameworks. In the past,
> I've mainly used the standard unittest framework built-in to Python to
> write tests. I was wondering if those of you familiar with nose or
> py.test could explain what these packages add to the standard
> framework? What is it that *we* need on top of the most basic testing
> functionality unittest provides?

Basically, both nose and py.test's biggest improvements over unittest
are much simpler testing syntax, and they gather and run whole test
suites more easily.

Compare the "basic example" section for unittest
(http://docs.python.org/library/unittest.html#basic-example) to the
basic examples for nose
(http://packages.python.org/nose/writing_tests.html).  Unittest
closely follows the JUnit technique, which is based on java-style
programming, and hence has a quite verbose syntax requiring a class
for basically every test, and yet another class if you want a whole
test suite.

With nose and py.test, on the other hand, you implement simply by
adding a "test_modname.py" file, and write functions in that file of
the form "def test_thisfeature()".  Then, when you run the command
line tools, it usually finds all your tests and runs them, issuing a
final report of passes and failures. (Also, "python setup.py test"
from distrbute/setuptools can be set to do the same thing, and it
seems to require quite a bit of configuration in unittest).

nose and py.test both also extend this functionality quite a bit and
provide additional bells and whistles, but this is really the main
advantage, in my view: they both can do everything unittest does (in
fact, by default they even call unittest tests), but with much simpler
syntax, and without all the (at least, to me) repetitive boilerplate
code you see in unittest.

> Since py.test and nose are so
> similar, what are the *differences*? I think if we know this, it will
> be easier for those of us not too familiar with either framework to
> weigh in on the issue.

They have mostly the same basic features, but a lot of the more
complicated tricks and the plugin libraries are slightly different in
their syntax and capabilities.  It appears that both nose and py.test
can run unittest and doctest tests, and py.test has explicit support
for a number of nose's unique features, but nose doesn't support
py.test syntax beyond where the two of them match.  In my mind this is
the best reason to pick py.test : it is the least restrictive because
people can continue writing tests in whatever they're used to, and
py.test should run it without problems.

Victoria pointed out that nose is apparently undergoing a development
hiccup as the main author is backing off, and the community is slowly
(which may mean never) picking up the slack.  py.test, on the other
hand, is currently being actively improved.  This may not matter too
much because both are already quite stable, but it might be a concern
for the future.   It appears both work on python 3, but it seems like
it has been more recently added for nose (it claims to be fully
supported, although 3rd party plugins are not necessarily).

After hearing from a few people (on-list and elsewhere) and glancing
over the docs, it looks to me that py.test is somewhat easier to write
plugins for than nose, partly because of a cleaner plugin architecture
(py.test is actually implemented internally as a bunch of plugins),
and more complete documentation.  This may be relevant for us because
we probably will need to do some of this if we want conformance
testing of some modules against "known-good" external libraries
(although py.test seems to have a builtin plugin that does that sort
of thing already?).

The only thing that I see as useful that only one of them has is that
py.test can generate a script that can be used to do all py.test
actions.  If we did theis, users don't even have to have py.test
installed!  So that would allow us to include the py.test runner
script in the source code, and people wouldn't need to install an
external package to collect/run the test suite like they do with nose.

In principle, the similarities between nose and py.test mean we
wouldn't really have to make a decision until we need features that
differ between them- developers could run whichever they prefer...
Except there's one problem: the configuration files are formatted
differently...  We want to use configuration settings for just one of
the packages so that the suite always runs the same.  So what we're
really deciding is which one to write configuration files for, and
request that people use as the "standard" test suite.

Given all the points here, I now think py.test is a mildly better
choice... but either py.test or nose would probably suit our needs.

> By the way, it looks like there's been a lot of work on unittest in
> python 2.7 (aka unittest2) so how does that compare to nose and
> py.test?

The biggest change I see is that it seems to now include a test
discovery tool, but it is much less documented (a few paragraphs
rather than many pages each for the other two), and seems to lack most
of the nose and py.test runners' customization (like configuring in
the setup.cfg file, easy plugins, etc.). It's an improvement, but it's
still based around the more complex syntax of unittest, so most of
what I said above as far as I can tell is still true.

Regardless, we couldn't use that anyway if we're planning to stay
2.6-compatible.

>
> Cheers,
> Tom
.+++
>
> On 13 July 2011 18:54, Erik Tollerud <erik.tollerud at gmail.com> wrote:
>>> b.t.w.  I suggest that you differentiate the _purpose_ of a test from
>>> the _mechanism_ of the test.
>>
>> This is a very good point - when I think about it more, it makes sense
>> to have the same test-runner framework.  With either nose or py.test
>> it seems reasonable to divide the testing into tests which always
>> depend only on the core modules, and a separate suite that is more
>> general, potentially using e.g. pysofa to test coordinate conversions
>> and the like.  As Mark says, there isn't necessarily a need (at least
>> at this stage) to carefully divide tests into categories, as long as
>> testing coverage is good for the actual purpose of the module.
>>
>>> What is "complete coverage" anyway - testing all public methods with all allowed input types (and possibly some that are not allowed to test the correct error handling)? This is probably not even realised for numpy right now, so I generally second that idea. In particular, it would probably needlessly delay acceptance of some largely finished packages that are otherwise ready for inclusion.
>>> I would probably put a somewhat stronger emphasis than "Unit tests are encouraged..." into the guidelines for any newly  developed or revised code, also to encourage test-driven development practices.
>>
>> Absolutely agreed - the plan is to make it clear in the example
>> package that everything should be unit-tested if it can be, as that's
>> what new packages will presumably use.
>>
>>
>> Now the nose vs. py.test question... From Mark's summary, it sounds
>> like it's difficult to choose between them because they're too
>> similar!  Given Mark and Victoria's comments (and looking over the
>> docs in more detail myself), I'm beginning to think that py.test has
>> more potential.  Especially important is that fact that it appears
>> that py.test can directly use tests written for nose, while the
>> converse is not true... this would make integrate already-existing
>> projects with nose much easier and lower the re-learning penalty to
>> pretty much nothing.  Anyone else have experience with py.test that
>> may be of relevance?
>>
>>
>>>
>>> Cheers,
>>>                                                        Derek
>>>
>>>> On Fri, Jul 8, 2011 at 12:57 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
>>>>>
>>>>>>> As for the testing framework, we were thinking nose
>>>>>>> (http://somethingaboutorange.com/mrl/projects/nose/1.0.0/).   Nose is
>>>>>>> very nice for a project like this because it's super easy to run and
>>>>>>> also easy to write tests with, but is also compatible with the stdlib
>>>>>>> unittests.
>>>>>
>>>>>
>>>>> "use nose" is a little ambiguous.  numpy "uses nose" but if you look
>>>>> inside numpy.test() it looks like of like they implemented their own
>>>>> testing system using nose as a library.  I've never succeeded in using
>>>>> nosetests to run the numpy tests, for example.
>>>>>
>>>>> If you choose nose, my preference is to say:  Tests come in a separate
>>>>> directory from the source.  You can run all the tests with the command
>>>>> "nosetests tests/".
>>>>>
>>>>> This is not to exclude the possibility of installing some of the tests
>>>>> or of providing a test() function, but I also want to run the tests
>>>>> directly with nosetests.
>>>>>
>>>>> I actually have an ulterior motive here:  I use a nose plugin that feeds
>>>>> a report generator.  If you only provide a foo.test() function, I can't
>>>>> necessarily get it to use the plugin even if you base the whole thing on
>>>>> nose.  You might not think that was a big deal, but it gets important
>>>>> when you have the test volume that I regularly deal with: > 100 000
>>>>> tests per night, sometimes > 10 000 fail, depending what the developers
>>>>> are up to.
>>>>>
>>>>> b.t.w.  This doesn't mean I have any attachment to nose; it just happens
>>>>> to be one of a small number of test runners that I use right now.
>>>>>
>>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>
>>
>>
>> --
>> Erik Tollerud
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>

-- 
Erik Tollerud