unittest.Testsuite and execution order

[Francesco Russo]

On 18/04/18 20:26, Chris Angelico wrote:
> On Thu, Apr 19, 2018 at 2:51 AM, Francesco Russo <francescor82 at gmail.com> wrote:
>> My use case: my SUT is split into modules. Besides writing unit tests for
>> each module, I want to write an integration test, and I also need to
>> perform some actions between two calls to the SUT. In my case, the order of
>> the execution is important.
> 
> In that case, make these tests into a single test. If you have to do
> the steps together to correctly test it, they're not separate tests,
> they're separate parts of the same test.

Clear, thank you.

>> class MyTestCode(unittest.TestCase):
>>    def test_func_1(self):
>>       # do something to test func_1 on the SUT
>>       sut.func_1()
>>       self.assert(...)
>>
>>    def perform_intermediate_step(self):
>>       # do something between func_1 and func_2
>>
>>    def test_func_2(self):
>>       # do something to test func_2 on the SUT
>>       sut.func_2()
>>       self.assert(...)
> 
> This is a bad idea. Each function that starts test_ should be
> completely independent. You should be able to run any one of them on
> its own (say, if you're trying to figure out why your latest change
> caused a test failure), and it should have the same result.
> 
> Make it so that test_func_1 and test_func_2 are completely
> independent, and then, if you need a single test that uses both, have
> a test that calls on each function.

I'm not sure I understand you here.
I understood that (besides, or instead of, making an integration test by
making those tests into one test, as you wrote above) I could make a
test for func_2 making it independent from func_1, for example this way:

class MyTestFunc2(unittest.TestCase):
   def setUp(self):
      # Prepare preconditions for func_2

   def test_func_2(self):
      sut.func_2()
      self.assert(...)

Such a test case wouldn't even need a test suite.
Is this what you meant?

>> Such an example works, today, since TestSuite uses a list, and addTest()
>> appends to the list.
>> My question is: is this something that I can rely on for the future? I
>> definitely don't want to rely on the current implementation, unless I see
>> it in the documentation.
> 
> I would say no, you can't rely on it. If you can't find it in the
> docs, don't assume it's true. Test order randomization can be
> controlled with a simple command line flag.
> 
> ChrisA

The official "unittest" web pages for Python 2 and 3 say this, for the
TestSuite class:
*This class represents an aggregation of individual tests cases and test
suites*
saying nothing about the order. But the docstring of the TestSuite class
says:
*It will run the individual test cases in the order in which they were
added, aggregating the results*
Can I consider the docstring an official documentation as well?

-- 
Francesco Russo
The White Rabbit put on his spectacles. 'Where shall I begin, please
your Majesty?' he asked.
'Begin at the beginning,' the King said gravely, 'and go on till you
come to the end: then stop.'