[Python-ideas] Adding a test discovery into Python

[Guido van Rossum]

On Sun, Feb 1, 2009 at 2:29 PM, Christian Heimes <lists at cheimes.de> wrote:
> Guilherme Polo schrieb:
>> So.. is there any chance we can enter in agreement what features would
>> be useful in a test discovery that could be included with Python ? I
>> for myself do not have fancy wishes for this one, I would be happy
>> with something that would collect unittests, inside packages and
>> subpackages, with some fixed patterns and let me run them with
>> test.test_support.run_unittests, or maybe something that would collect
>> unittests and doctests and have something like run_tests in
>> test.test_support. But then I believe this wouldn't be good enough to
>> substitute any of the current tools, making the addition mostly
>> useless.
>
> I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple
> because there are enough frameworks for elaborate unit testing.
>
> Such a tool should
>
> - find all modules and packages named 'tests' for a given package name

I predict that this part is where you'll have a hard time getting
consensus. There are lots of different naming conventions. It would be
nice if people could use the new discovery feature without having to
move all their tests around.

> - load all subclasses of unittest.TestCase from 'tests' module or
>  '*/tests/test*.py' files

Once you've found the test modules, TestCase subclasses are a good
place to start. Though beware -- there's a pattern that defines
several alternative classes in a module, e.g. one per platform, and
then defines a test suite that dynamically decides which class to use.
The stdlib test suite uses this in a number of places, though I can't
quite remember where.

> - support some basic filtering for test cases or test functions

Or module names.

> Do we need more features?

I'd look at some of the popular alternatives to unittest.py and see
what wisdom or conventions they have to offer.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)