[issue25894] unittest subTest failure causes result to be omitted from listing

[Serhiy Storchaka]

Serhiy Storchaka <storchaka+cpython at gmail.com> added the comment:

PR 28082 is a draft that implements this idea. Skipped and failed (but not successfully passed) subtests are now reported separately, as a character (sFE) or a line ("skipped", "FAIL", "ERROR"). The description of the subtest is included in the line. For example:

$ tests=.sFE ./python test_issue25894.py 
sFE
======================================================================
ERROR: test_subTest (__main__.TestClass) [3] (t='E')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/test_issue25894.py", line 15, in test_subTest
    raise Exception('error')
    ^^^^^^^^^^^^^^^^^^^^^^^^
Exception: error

======================================================================
FAIL: test_subTest (__main__.TestClass) [2] (t='F')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/test_issue25894.py", line 13, in test_subTest
    self.fail('failed')
    ^^^^^^^^^^^^^^^^^^^
AssertionError: failed

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1, errors=1, skipped=1)

$ tests=.sFE ./python test_issue25894.py -v
test_subTest (__main__.TestClass) ... 
  test_subTest (__main__.TestClass) [1] (t='s') ... skipped 'skipped'
  test_subTest (__main__.TestClass) [2] (t='F') ... FAIL
  test_subTest (__main__.TestClass) [3] (t='E') ... ERROR

======================================================================
ERROR: test_subTest (__main__.TestClass) [3] (t='E')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/test_issue25894.py", line 15, in test_subTest
    raise Exception('error')
    ^^^^^^^^^^^^^^^^^^^^^^^^
Exception: error

======================================================================
FAIL: test_subTest (__main__.TestClass) [2] (t='F')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/test_issue25894.py", line 13, in test_subTest
    self.fail('failed')
    ^^^^^^^^^^^^^^^^^^^
AssertionError: failed

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1, errors=1, skipped=1)

As a side effect, the test description is also repeated for every error in the test cleanup code (in tearDown() and doCleanup()).

Similar changes should be added also in RegressionTestResult. If apply issue45057 first they will be much simpler.

Issue29152 can be related. If call addError() and addFailure() from addSubTest(), PR 28082 should be rewritten.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue25894>
_______________________________________