[issue22197] Allow better verbosity / output control in test cases

[Robert Collins]

Robert Collins added the comment:

There's a few interacting things here. If I can suggest some design thoughts.

buffering within a test is I think really something we should offer a test servicing API for. There are many thirdparty ones (e.g. I have one in fixtures) - but it should be a dedicated API, even if the implementation happens to be shared with the buffer runner feature.

I think it is reasonable in some cases to change TestCase behaviour based on how much debugging is desired - even in CI environments, some outputs are much more expensive than others to generate (e.g. taking a full dump of a database). To enable that we do need a CLI -> runtime environment communication mechanism.

There are many related APIs which we could look at for doing this.
* TestCase's load API (__init__) - we'd get at this via TestLoader
* TestCase's execution API (run, debug) - we'd get at this through TestRunner (for more parameters) or perhaps TestResult (by adding some attribute/query method).
* TestCase's test servicing API (assert*) - we could add a query method or a well known attribute to abstract out where the info is actually coming from
* TestCase's user supplied API (setUp, test methods)
* TestResult - constructed by the runner, it is in some ways the natural means for passing stuff like this around - its how result does its buffering today: but IMO the implementation is in the wrong spot there [it precludes parallelisation vs a test service API that uses a mutex only when needed], so I'd like to not repeat that here.

TestCases support being run multiple times, so to me that rules out using the load API.

I very much want to take TestResult in the direction of being forward-only: feed-backwards APIs on it are much more complex to reason about in the case of Y joins in the result chain. So I think its time we introduced a runner -> test API.

Something like

def run(self, result=None, runner=None):
   ...
   dunder_set(self, '__unittest__runner__', runner)
   try:
   ...
   finally:
       dunder_set(self, '__unittest__runner__', None)
   
def debug(self, runner=None):
   ... same ...

with a helper to pull that out without folk's own code triggering name mangling.

And runner would then be the home for things like aborting a test run, providing configuration like verbosity, arbitrary end user options and so on.
Something like:
runner.config = {}
runner.config['verbosity'] = ...
runner.config['user'] = {... marshalled from CLI ... }

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22197>
_______________________________________