Unittests and serial workflows

[Steven D'Aprano]

On Tue, 27 Oct 2015 03:10 pm, Fabien wrote:

> Hi all,
> 
> I am not trained as a software developer and have only superficial
> knowledge about the theories and good practices of software testing.
> 
> Say I have several tasks (functions) A(), B(), C(), etc., each of them
> depending on the previous one. My understanding of unit testing is that
> each function should be tested alone and each test should be
> independent, 

Yes, that's the standard definition of unit testing.

However, unit testing frameworks are often perfectly well suited to other
forms of tests, like regression tests, integration tests, and others.
Sometimes it is perfectly reasonable to write a test which, strictly
speaking, doesn't fall under the unit test definition.



> so that currently my tests look like this: 
> 
> class TestTasks(unittest.TestCase):
> 
>      def test_a(self):
>          out = A()
>          self.assertStuffs(out)
> 
>      def test_b(self):
>          tmp = A()
>          out = B(tmp)
>          self.assertStuffs(out)

That's fine and meets the definition of a unit tests. Unit tests are allowed
to assume that other components of the module are working.

In test_b, you assume that A() is working correctly, and test B(). If A() is
not in fact working correctly, then the test may be invalid, or a bug in
A() may cause cascading failures: test_a will hopefully fail, and test_b
then fail as well.

Sometimes you want to avoid such cascading failures. To do that, rather than
call A() inside test_b, you create a simulation of A(), or hard-code a
value that could have been generated by A(). Such things are often called
mocks and stubs. Something like this perhaps:

    def test_a(self):
        out = A()
        self.assertEqual(out, (1, 2, 3, 4))

    def test_b(self):
        tmp = (1, 2, 3, 4)
        out = B(tmp)
        self.assertStuffs(out)


Now, even if A() is buggy and returns something invalid, the argument to B()
is fine and you can test B() in isolation to A().

 
> And so forth. The order in which unittest is running the test is not the
> one given by the code (it seems to be alphabetical?) so that I am not
> guaranteed that my functions are tested in the right order (my real
> tasks of course are not sorted alphabetically).

In my opinion, if your unit tests need to be run in a specific order, they
aren't sufficiently isolated. But, I accept that sometimes reality bites,
and you have to deal with tests which aren't sufficiently isolated. You
don't have time to fix the tests, and you can't fix the code you're
testing. Sucks to be you... 

It's times like that you want to use a *subclass* of unittest which gives
you a predictable order. Have fun writing one :-)


> Furthermore, it is inefficient since task A() is going to be run many
> times. 

You shouldn't care (too much) about the efficiency of tests. You don't want
them to be *horribly* inefficient, of course, but it it takes 30 seconds to
run the tests, who cares? Even a minute or so is no big deal.

A big test suite like that for Python itself, with dozens of modules to be
tested, might take ten minutes to run, just long enough for you to stretch
your legs, get the cobwebs out of your brain, and make a coffee.

Of course you need the ability to run individual test suites, for those
rapid debug/run tests/debug cycles. But don't worry about optimizing each
test. That's a waste of time, and likely to introduce bugs into the tests
themselves. Write your tests in the simplest, most fool-proof way you can,
and don't be too clever.


> Of course I could run all the tasks once, store the results and 
> run my tests afterwards one by one but this contradicts somehow the idea
> of unit testing. And I like to be able to run each test alone (in
> PyCharm a mouse click alone starts a single test).
> 
> My question(s): is Unittest the "right" framework for my needs? Should I
> move to Nosetests? Any hint to a useful, not too long document about
> testing framework and good practices?

http://thedailywtf.com/articles/Testing-Done-Right

And of course, read the Daily WTF for plenty of examples of what not to do.
They have plenty of examples of bad testing.




-- 
Steven