[Python-ideas] Test Class setup and teardown in unittest

[Robert Collins]

On Thu, 2010-01-21 at 11:42 +0000, Michael Foord wrote:
> On 21/01/2010 01:37, Robert Collins wrote:
> > On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
> >
> >> Earlier this week on the Testing In Python list, there was a
> >> discussion on how to execute a setup and/or teardown for a single test
> >> class instead of for each test fixture on the class (see the 'setUp
> >> and tearDown behavior' thread).  I have had to deal with situation
> >> myself before, and I am obviously not the only one (since I did not
> >> initiate the thread).  As such I'd like to propose adding a class
> >> level setup and tear down method the unittest TestCase class.
> >>
> > I think that this is the wrong approach to the problem:
> >   - class scope for such fixtures drives large test classes, reduces
> > flexability
> >
> 
> Agreed.
> 
> >   - doing it the rough way you suggest interacts in non obvious ways with
> > test order randomisation
> >
> 
> Not currently in unittest, so not really an issue.

Done by nearly every extended UI out there. I would hope that messing
them up would count as an issue.

> >   - it also interacts with concurrent testing by having shared objects
> > (classes) hold state in a way that is not introspectable by the test
> > running framework.
> >
> 
> Again, unittest doesn't do concurrent testing out of the box.

testtools includes a concurrent TestResult which buffers activities from
different executing test cases and outputs them to a single regular
TestResult. Using that run() on TestSuite can be redefined to start some
threads, giving each one a helper TestResult forwarding to the caller
supplied TestResult, and finally treat self.__iter__ as a queue to
dispatch to each worker thread.

> How are shared fixtures like this not introspectable?

How can you tell if there is shared state when setUpClass is being used?

> As I see it.
> 
> Advantages of setupClass and teardownClass:
> 
> * Simple to implement
> * Simple to explain
> * A commonly requested feature
> * Provided by other test frameworks in and outside of Python

Mmm. It is commonly requested, but I'm not sure I entirely agree about
the simple claims.

> Disadvantages:
> 
> * Can encourage a poor style of testing, including monolithic test classes

* Can limit concurrency

> > I'd much much much rather see e.g. testresources integrated into the
> >
> 
> How is testresources superior? 

It separates concerns, which allows introspection for test runners. This
is significantly more flexible, particularly when combined with test
parameterisation.

> Can you demonstrate this - in particular 
> what is the simplest example? The simplest example of setupClass would be:
> 
> class SomeTest(unittest.TestCase):
>      def setupClass(self):
>          ... # setup shared fixture

                     ^ this is buggy :)
Its buggy because if it really does write to self, you now have what
twisted did where all the test cases will be sharing one test object
instance - which very frequently confuses and makes isolation between
unrelated test state fragile.

If its not writing to self, then you don't know what class object to
write to (because you might be in a subclass) - that really should be a
class method.

And the fact that you have this confusion, as demonstrated here, is
precisely why I don't agree about easy to implement and easy to
understand.

> > core allowing fixtures to be shared across test instances in a way that
> > doesn't prohibit their use with concurrent testing, doesn't make it
> > awkward to do it across multiple classes.
> 
> Can you demonstrate how testresources solves these problems.

Yes. resources are declared in a list, which allows the test framework,
when parallelising, to group together tests that use the same resources
when partitioning. Compare this with setUpClass where all tests in an
inheritance hierarchy have to be grouped together (after excluding your
base TestCase).

You ask in another mail for a SSCE: here is one attached to this mail
('foo.py'). Run with 'python -m unittest foo'.

> > I'm happy to make any
> > [reasonable] changes (including license) to testresources to make it
> > includable in the stdlib if that's of interest.
> >
> 
> That's not how contributing code to Python works. You need to provide a 
> signed contributor agreement to the PSF and then you specifically 
> license to the PSF any code you are contributing using one of a few 
> specific licenses. For new modules in the standard library you also need 
> to be willing to maintain the code.

There is other code of mine in the standard library unittest module,
done without a contributor agreement: so this is at best inconsistent. I
mention license changes because that would be required to include it in
the standard library (its currently a copyleft license) - and I'm not
the sole author. Regardless, whatever process and actions are needed, I
will do them, or arrange with other people where needed.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.py
Type: text/x-python
Size: 546 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100122/5658864a/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100122/5658864a/attachment.pgp>