[Python-Dev] setUpClass and setUpModule in unittest

[Michael Foord]

On 15/02/2010 17:05, Michael Foord wrote:
> [snip]
> This is also an interesting point. The 'naive' implementation, which I 
> think I prefer, only runs the setUpModule of modules actually 
> containing tests. Similarly setUpClass is only called on classes with 
> actual tests, although they may call up to base class implementations.
>
> This has a couple of consequences, particularly for setUpModule. It 
> makes the basic rule:
>
> * only use setUpModule for modules actually containing tests
> * don't mix concrete TestCases (with tests) in the same module (using 
> setUpModule) as base classes for other tests
>
> The use case (that I can think of) that isn't supported (and of course 
> all this needs to be documented):
>
> * Having setUpModule / tearDownModule in modules that define base 
> classes and expecting them to be called around all tests that inherit 
> from those base classes
>
> Having this in place makes the implementation simpler. If we 
> explicitly document that this rule may change and so users shouldn't 
> rely on setUpModule not being called for modules containing base 
> classes, then we are free to rethink it later. Not having this 
> restriction at all is possible, it just requires more introspection at 
> TestSuite creation / ordering time.
>
> Note that setUpClass on a base class maybe called several times if 
> several base classes inherit and all call up to the base class 
> implementation. As it will be a class method the cls argument will be 
> different for each call.
>
> Another question. If we are implementing TestCase.setUpClass as an 
> additional test then should it be reported *even* if it is only the 
> default (empty) implementation that is used?
>
> The reason to have setUpClass implemented as a test is so that you can 
> report the failure *before* you run all the tests. Lots of unit test 
> users want a consistent number of tests every run - so we shouldn't 
> insert an extra test only on fail. The other alternative is to report 
> a setUpClass failure as part of the first test that depends on it - 
> this makes the implementation more, complex (having a pseudo-test 
> represent setUpClass / tearDownClass is convenient for ordering and 
> isolating the 'magic' in one place - the rest of the test running 
> infrastructure doesn't need to know about setUpClass or Module).
>
> If we do add a default setUpClass test for all TestCases it means 
> extra noise for test runs that don't use the new feature. Saying no it 
> shouldn't be shown means that we have to introspect test classes to 
> see if they inherit setUpClass from TestCase or from some intermediate 
> base class. Not hard just an extra complexity. One solution would be 
> for TestCase *not* to have default implementations, but it is probably 
> nicer for them to appear in the API.
>
> I guess my preferred approach is to have a default implementation, but 
> not to create pseudo-tests for them if they aren't used.
>

One place to implement this is in the TestLoader (specifically in 
loadTestsFromModule and loadTestsFromTestCase) - especially as this is 
the place where test ordering is currently provided by unittest. The 
TestSuite may not need to change much. This isn't compatible with test 
frameworks that build custom suites without going through 
loadTestsFromModule - e.g. modules implementing load_tests that replace 
the standard test suite with a new suite with specific TestCases and 
still expect setUpModule / tearDownModule to be used [1]. Perhaps an API 
hook to make this easy?

TestCase would need a change in run(...) to support automatic 
SetupFailed test failing when setUpClass / module fails.

This *isn't* compatible with custom TestSuites that reorder tests. The 
alternative is for TestSuite.run(...) to change to support setUpClass / 
setUpModule by adding them at test run time by introspecting contained 
tests. That would make setUpClass (etc) incompatible with custom 
TestSuite implementations that override run. It would be compatible with 
TestSuites that do reordering but that don't overload run(...) - 
although the resulting test order may not be 'optimal' (the setUp and 
tearDown may end up separated farther in time than desired but at least 
they would work). Perhaps this approach is less likely to break? 
(Manually reordering tests or adding new tests to a TestSuite wouldn't 
break setUpClass as the TestSuite only adds them at run(...) time rather 
than placing them in the suite where the 'user' can move them around).

All the best,

Michael

[1] or even load_tests that just addTests to the standard suite but 
still need tearDownModule to be run *after* all the tests. Even with the 
implementation in loadTestsFrom* it seems like the TestSuite may need to 
know something about the order. Perhaps we could have 
TestLoader.orderTests - the loader is available to load_tests functions.


>
> All the best,
>
> Michael
>
>> testresources very neatly sidesteps this problem by just providing an 
>> API to say "this test case depends on that test resource", without 
>> relying on the grouping of tests within classes, modules, or 
>> packages.  Of course you can just define a class-level or 
>> module-level resource and then have all your tests depend on it, 
>> which gives you the behavior of setUpClass and setUpModule in a more 
>> general way.
>>
>> -glyph
>>
>
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.