[Python-Dev] setUpClass and setUpModule in unittest

[Glyph Lefkowitz]

On Feb 15, 2010, at 3:50 PM, Michael Foord wrote:

> On 15/02/2010 20:27, Glyph Lefkowitz wrote:
>> 
>> 
>> On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:
>> 
>>> On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
>>> <glyph at twistedmatrix.com> wrote:
>>>> I find setUpClass more hostile to *other* kinds of testing, because this convenience for simple integration tests makes more involved, performance-intensive integration tests harder to write and manage.
>>> 
>>> That sounds odd, as if the presence of this convenience would prohibit
>>> you from also implement other features.
>> 
>> And I am pretty sure this is not just my over-reaction; Michael still appears to be wrestling with the problems I'm describing.
> And I appreciate your input.

Thanks :).

>>  In a recent message he was talking about either breaking compatibility with TestSuite implementations that override run(), or test-reordering - both of which I consider important, core features of the unittest module.
> 
> Well, by "breaking compatibility with custom TestSuite implementations that override run" I mean that is one possible place to put the functionality. Code that does override it will *not* stop working, it just won't support the new features.

Ah, I see.  This doesn't sound *too* bad, but I'd personally prefer it if the distinction were a bit more clearly drawn.  I'd like frameworks to be able to implement extension functionality without having to first stub out functionality.  In other words, if I want a test suite without setUpClass, I'd prefer to avoid having an abstraction inversion.

Practically speaking this could be implemented by having a very spare, basic TestSuite base class and ClassSuite/ModuleSuite subclasses which implement the setUpXXX functionality.

> If we chose this implementation strategy there would be no compatibility issues for existing tests / frameworks that don't use the new features.

That's very good to hear.

> If tests do want to use the new features then the framework authors will need to ensure they are compatible with them. This seems like a reasonable trade-off to me. We can ensure that it is easy to write custom TestSuite objects that work with earlier versions of unittest but are also compatible with setUpClass in 2.7 (and document the recipe - although I expect it will just mean that TestSuite.run should call a single method if it exists).

This is something that I hope Jonathan Lange or Robert Collins will chime in to comment on: expanding the protocol between suite and test is an area which is fraught with peril, but it seems like it's something that test framework authors always want to do.  (Personally, *I* really want to do it because I want to be able to run things asynchronously, so the semantics of 'run()' need to change pretty dramatically to support that...)  It might be good to eventually develop a general mechanism for this, rather than building up an ad-hoc list of test-feature compatibility recipes which involve a list of if hasattr(...): foo(); checks in every suite implementation.

> Perhaps a better idea might be to also add startTest and stopTest methods to TestSuite so that frameworks can build in features like timing tests (etc) without having to override run itself. This is already possible in the TestResult of course, which is a more common extensibility point in *my* experience.

I think timing and monitoring tests can mostly be done in the TestResult class; those were bad examples.  There's stuff like synthesizing arguments for test methods, or deciding to repeat a potentially flaky test method before reporting a failure, which are not possible to do from the result.  I'm not sure that startTest and stopTest hooks help with those features, the ones which really need suites; it would seem it mostly gives you a hook to do stuff that could already be done in TestResult anyway.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100216/de3ecf72/attachment.htm>