[Python-ideas] Test Class setup and teardown in unittest

[Robert Collins]

On Wed, 2010-01-20 at 22:33 -0500, Mark Roddy wrote:


> > I'd take issue with the argument that this makes the intent clearer. In your
> > motivating example, what you mean is "the test needs a connection" not "I want
> > to do a set up that spans the scope of this class". A better approach would be
> > to _declare_ the dependency in the test and let something else figure out how to
> > best provide the dependency.
> I think having the class setup is clearer than the 'if not
> self.CaseIsSetup' idiom, but I wouldn't claim that it's definitely the
> clearest way to achieve such functionality.
> 
> I like the class setup semantics as the closely follow the existing
> fixture setup semantics.  Having to declare and configure dependencies
> (while possibly being clearer as far as expressing intent) introduces
> a completely different set of semantics that have to be grasped in
> order to be used.  I'm going to with hold forming an opinion as to
> whether or not this presents any meaningful barrier to entry until
> after I've had a chance to review Rob's resources library.

A data point here is that the setUp/tearDown/cleanUp semantics are not
trivial themselves: here is a self check (check the source after coming
up with answers :P)
 - does tearDown run if setUp raises?
 - do cleanUps?
 - do cleanUps run before or after tearDown?

Also would you add classCleanUps or something, if you add a class setUp?
clean ups are _much_ nicer than tearDown, so I'd really want to see
that.

> > It turns out that if you are building extensions to testing frameworks (like any
> > big app does), it helps a lot to have an object per runnable test. In
> > particular, it makes distributing tests across multiple processors & multiple
> > computers much, much easier.

> True, and clearly the case for highly focused unit tests.  However,
> for integration tests (or whatever we could call tests that are
> explicitly designed to work with an expensive resource) it can be cost
> prohibitive to have this type of isolation (or flat out impossible in
> the case that I gave).

Its been trimmed, but here was your example:
"For
example, I have several hundred integration tests that hit a live
database.  If I create the connection object for each fixture, most of
the tests fail due to the maximum connection limit being reached.  As
a work around, I create a connection object once for each test case."

Database connections can be dirtied - they need to be in a clean state
with no outstanding requests, and outside a transaction, to be valid for
the start of a test. I do appreciate that if you have a buggy TCP stack,
or are not closing your connections, you can certainly hit connection
limits. 

I'd be inclined to have a pool of connections, rather than one
connection per class. That would under ideal circumstances give you one
connection for an entire test run (without concurrency), but also mean
that a broken connection would at most affect only one test.

>   I'll look into the implementation of some of
> the testing frameworks that support distributed testing and see if
> there isn't a way that this can be supported in both contexts (is it
> possible to implement in a way that the case setup would get run in
> each process/machine?).

Its about partitioning the state: e.g. (ugly) deep copy the class object
per thread.

> > It also poses a difficult challenge for test runners that provide features such
> > as running the tests in a random order. It's very hard to know if the class is
> > actually "done" and ready to be torn down.
> My initial (off the top of my head) thinking was to count the number
> of test fixtures to be run via a class attribute set in the test case
> constructor, and then the test runner would either decrement this
> after the test is complete and call the tear down method once the
> counter reached zero.  This doesn't seem like it would be affected by
> randomized testing ordering, but I'll look into some existing
> implementations to see how this could be affected.  Any obvious issues
> I'm missing?

Well, this has a few undesirable facets:
 - its complex
 - you'll need to honour tests that don't have that attribute set
 - and tests that already have that attribute set
 - it breaks the abstraction of 'a test knows how to run itself'.
 - you can't set that attribute in test case constructors because tests
can be constructed while the suite is running
 - using a global like that will mean that a single classes class setup
stuff would *stay setup* while other classes run, with a randomised
order : thats bad because things that are expensive and need to be
optimised also tend to be *large*, either in memory or external
resources (like TCP connections in your example).

> > Finally, we found that it's use often lead to hard-to-debug failures due to test
> > isolation issues.

> I think there's a distinction between "can lead to bad situations" and
> "encourages bad situations".  The former is almost impossible to avoid
> without becoming java :).  That latter is much subtler, but can be
> addressed.  Do you have any suggestions for altering the semantics to
> discourage abuse without reducing flexibility.  With a similar feature
> I use, we have a rule to not use the case setup unless explicitly
> writing integration tests, though there is no functional way to
> enforce this, only communicating the idea (via documentation and code
> reviews).

Have you seen Rusty Russell's interface usability scale? Having to have
a rule strongly suggests that the API is prone to misuse :).

> > I'm not deeply familiar with xUnit implementations in other languages, but the
> > problem isn't unique to Python. I imagine it would be worth doing some research
> > on what Nunit, JUnit etc do.
> Both JUnit and Nunit have a class setup and teardown functionality:
> http://www.junit.org/apidocs/org/junit/BeforeClass.html
> http://www.junit.org/apidocs/org/junit/AfterClass.html
> http://www.nunit.org/index.php?p=fixtureSetup&r=2.5.3
> http://www.nunit.org/index.php?p=fixtureTeardown&r=2.5.3
> (The Nunit implementation I found a little confusing as they
> apparently refer to a TestCase as a Fixture).

See my other mail for a link to JUnit's Rules.

Cheers,
Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100121/e14b4c05/attachment.pgp>