[Python-ideas] Adding a test discovery into Python

[Brett Cannon]

On Sun, Feb 1, 2009 at 14:40, Guido van Rossum <guido at python.org> wrote:
> On Sun, Feb 1, 2009 at 2:29 PM, Christian Heimes <lists at cheimes.de> wrote:
>> Guilherme Polo schrieb:
>>> So.. is there any chance we can enter in agreement what features would
>>> be useful in a test discovery that could be included with Python ? I
>>> for myself do not have fancy wishes for this one, I would be happy
>>> with something that would collect unittests, inside packages and
>>> subpackages, with some fixed patterns and let me run them with
>>> test.test_support.run_unittests, or maybe something that would collect
>>> unittests and doctests and have something like run_tests in
>>> test.test_support. But then I believe this wouldn't be good enough to
>>> substitute any of the current tools, making the addition mostly
>>> useless.
>>
>> I'm +1 for a simple (!) test discovery system. I'm emphasizing on simple
>> because there are enough frameworks for elaborate unit testing.
>>
>> Such a tool should
>>
>> - find all modules and packages named 'tests' for a given package name
>
> I predict that this part is where you'll have a hard time getting
> consensus. There are lots of different naming conventions. It would be
> nice if people could use the new discovery feature without having to
> move all their tests around.
>

Guido is right; this cannot be convention-based but instead
configuration-based. Let me specify in a function call what package to
look in and what naming convention I used for test files.

>> - load all subclasses of unittest.TestCase from 'tests' module or
>>  '*/tests/test*.py' files
>
> Once you've found the test modules, TestCase subclasses are a good
> place to start. Though beware -- there's a pattern that defines
> several alternative classes in a module, e.g. one per platform, and
> then defines a test suite that dynamically decides which class to use.
> The stdlib test suite uses this in a number of places, though I can't
> quite remember where.
>

Really? I have never come across that before in the standard library,
but I am sure people do stuff like that.

There is no way any tool will work in all pre-existing situations like
this where people heavily rely on a test_main function to handle
decisions on what to call. But as long as it is made clear how to
restrict what tests are found and it is an easy solution I am sure
people will adjust.

>> - support some basic filtering for test cases or test functions
>
> Or module names.
>

yes please! To be able to specify only the test being worked on would
be very nice indeed.

-Brett