[issue16997] subtests

[Chris Jerdonek]

Chris Jerdonek added the comment:

Nice/elegant idea.  A couple comments:

(1) What will be the semantics of TestCase/subtest failures?  Currently, it looks like each subtest failure registers as an additional failure, meaning that the number of test failures can exceed the number of test cases.  For example (with a single test case with 2 subtests):

$ ./python.exe test_subtest.py 
FF
======================================================================
FAIL: test (__main__.MyTests) (i=0)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subtest.py", line 9, in test
    self.assertEqual(0, 1)
AssertionError: 0 != 1

======================================================================
FAIL: test (__main__.MyTests) (i=1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subtest.py", line 9, in test
    self.assertEqual(0, 1)
AssertionError: 0 != 1

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=2)

With the way I understand it, it seems like a subtest failure should register as a failure of the TestCase as a whole, unless the subtests should be enumerated and considered tests in their own right (in which case the total test count should reflect this).

(2) Related to (1), it doesn't seem like decorators like expectedFailure are being handled correctly.  For example:

    @unittest.expectedFailure
    def test(self):
        for i in range(2):
            with self.subTest(i=i):
                self.assertEqual(i, 0)

This results in:

    Ran 1 test in 0.001s

    FAILED (failures=1, unexpected successes=1)

In other words, it seems like the decorator is being applied to each subtest as opposed to the test case as a whole (though actually, I think the first should read "expected failures=1").  It seems like one subtest failing should qualify as an expected failure, or are the semantics such that expectedFailure means that every subtest must fail?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16997>
_______________________________________