[Python-Dev] Disable tests in unittest (issue3202)

[Justin Mazzola Paluska]

On Thu, Jun 26, 2008 at 10:51:49AM +1000, Jonathan Lange wrote:
> On Thu, Jun 26, 2008 at 7:13 AM, Justin Mazzola Paluska <jmp at mit.edu> wrote:
> I think it's really worth looking at the approach that bzrlib's tests
> take here (see bzrlib.tests.ExtendedTestResult and the
> out-of-date-but-helpful http://bazaar-vcs.org/BzrExtendTestSuite)
> 
> Instead of disabling the tests, their practice is to run the tests but
> have them raise KnownFailure. When they write a test that they know
> will fail, they have it raise this exception. The extended TestResult
> object catches this exception in addFailure and reports it
> appropriately.[1]

I wasn’t aware of bzrlib’s extentions to unittest — many of them,
including KnownFailure, TestSkipped,
bzrlib.test.ExtendedTestResult.expectFailure, and the sundry list of
bzrlib.test.TestCase.assert* look useful enough to belong in the
standard library.

<snip — example>
> I can see that disabling still has some value, but I'd rather have
> KnownFailure first.

I think disabling and the bzr extensions have different use cases.
KnownFailure lets someone with an intimate knowledge of the the
failing test case write down that they expect the test to fail.  The
disabled decorator lets someone less familiar with the test cases to
easily disable tests that are getting in their way, without needing to
understand the body of the test function (or worse, mis-understanding
it and breaking it in a hard-to-understand way).

I agree that in the ideal world, we’d like to have both.  In fact, if
we had the bzr extensions, it’d be easy to write the disabled
decorator in terms of their KnownFailure:

def disabled(func):
    @wraps(func):
    def wrapper(*args, **kwargs):
        raise KnownFailure("%s disabled" % func.__name__)
    return wrapper

However, unless Python’s unittest is going to incorporate the useful
features of bzrlib’s TestCase object[*], it probably makes sense to
implement the disabled feature separately.
	—Justin

[*] Perhaps in the long term, this is the right thing to do as
unittest seems to have fallen behind in useful feature count as its
“conceptual” parent JUnit.