[Tutor] Questions as to how to run the same unit test multiple times on varying input data.

[Steven D'Aprano]

On Sat, Sep 24, 2016 at 12:55:28AM -0500, boB Stepp wrote:

> "Write a function named right_justify that takes a string named s as a
> parameter and prints the string with enough leading spaces so that the
> last letter of the string is in column 70 of the display.  Hint:  Use
> concatenation and repetition..."

Or the built-in str.rjust() method... :-)

> The problem is pretty easy, but the point is for me is to practice the
> steps of TDD and learn how to use unit testing, and I figure using
> easy problems as I go through the book will make learning TDD and
> testing easier.  After a couple of iterations I have this program,
> right_justify.py:
> 
> #!/usr/bin/env python3
> 
> '''Exerise 3.1 from "Think Python 2" by Allen Downey.
> 
> This module will take a string and right justify it so that the last character
> of the line will fall in column 70 of the display.  This is accomplished by
> padding the line with spaces.'''
> 
> def right_justify(a_string):
>     '''This fucntion will take the string, "a_string", and left justify it by

Left justify?

>     padding it with spaces until its last character falls in column 70 of the
>     display.  The padded string will be returned.'''
> 
>     return ' ' * 58 + a_string
> 
> Of course, this code passed its test when the string was "Monty
> Python", so I next wanted to run the same test with different strings
> each time, so that my code would fail in these more general cases.
> However, I struggled to find a way to repeat the same test with
> varying data,

There are two simple ways to handle testing with different test cases:

- write multiple test cases, one of each test;
- write a multiple tests within a single test case.

Of course these are not mutually exclusive. You can do both.


For example:

# Untested.
from my_module import right_justify
import random

class RightJustifyTest(unittest.TestCase):
    def test_right_justify(self):
        self.assertEqual(
           right_justify('Monty Python'),
           '                                                          Monty Python'
           )
        self.assertEqual(
           right_justify('NOBODY EXPECTS THE SPANISH INQUISITION!!!'),
           '                             NOBODY EXPECTS THE SPANISH INQUISITION!!!'
           )

    def test_empty_string(self):
        self.assertEqual(right_justify(''), ' '*70)

    def test_single_char(self):
        for c in 'abcdefg':
            with self.subTest(c=c):
                self.assertEqual(right_justify(c), ' '*69+c)

    def test_issue12345(self):
        # Regression test for issue #12345: right_justify() of a 
        # single space returns the empty string.
        input = ' '
        assert len(input) == 1 and ord(input) == 32
        expected = ' '*70
        self.assertEqual(right_justify(input), expected)

    def test_random_string(self):
        input = []
        for i in range(random.randint(2, 69):
            input.append(random.choice('abcdefghijklmnopqrstuvwxyz'))
        input = ''.join(input)
        if random.random() > 0.5:
            input = input.upper()
        expected = ' '*(70-len(input)) + input
        assert len(expected) == 70
        self.assertEqual(right_justify(input), expected)



> class TestRightJustify(unittest.TestCase):
[...]
>         self.test_strings = [
>                 "Monty Python",
>                 "She's a witch!",
>                 "Beware of the rabbit!!!  She is a vicious beast who will rip"
>                 " your throat out!"]

I'm not sure that you should name your data "test something". unittest 
takes methods starting with "test" as test cases. I'm reasonably sure 
that it can cope with non-methods starting with "test", and won't get 
confused, but I'm not sure that it won't confuse *me*. So I would prefer 
to use a different name.


>     def test_returned_len_is_70(self):
>         '''Check that the string returned by "right_justify(a_string)" is the
>         length of the entire line, i.e., 70 columns.'''
> 
>         for test_string in self.test_strings:
>             with self.subTest(test_string = test_string):
>                 length_of_returned_string = (
>                     len(right_justify.right_justify(test_string)))
>                 print('The current test string is: ', test_string)
>                 self.assertEqual(length_of_returned_string, 70)


It is probably not a good idea to use print in the middle of 
unit testing. The unittest module prints a neat progress display, and it 
would be rude to muck that up with a print. If you really need to 
monitor the internal state of your test cases, write to a log file and 
watch that.

Of course, you can *temporarily* insert a print into the test suite just 
to see what's going on. Test code is code, which means it too can be 
buggy, and sometimes you need to *debug your tests*. And we know that a 
simple but effective way to debug code is using temporary calls to 
print.

In this specific case, you probably don't really case about the test 
cases that pass. You only want to see *failed* tests. That's what the 
subTest() call does: if the test fails, it will print out the value of 
"test_string" so you can see which input strings failed. No need for 
print.

> Anyway, running the tests gave me this result:
> 
> c:\thinkpython2\ch3\ex_3-1>py -m unittest
> The current test string is:  Monty Python
> The current test string is:  She's a witch!
> The current test string is:  Beware of the rabbit!!!  She is a vicious
> beast who will rip your throat out!


/me looks carefully at that output...

I'm not seeing the unittest progress display, which I would expect to 
contain at least one F (for Failed test). Normally unittest prints out a 
dot . for each passed test, F for failured tests, E for errors, and S 
for tests which are skipped. I do not see them.


> ======================================================================
> FAIL: test_returned_len_is_70 (test_right_justify.TestRightJustify)
> (test_string="She's a witch!")
> Check that the string returned by "right_justify(a_string)" is the
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "c:\thinkpython2\ch3\ex_3-1\test_right_justify.py", line 32, in
> test_returned_len_is_70
>     self.assertEqual(length_of_returned_string, 70)
> AssertionError: 72 != 70


Notice that unittest prints the name of the test, the value or values 
you pass to subTest, a description of the test (taken from the doc 
string of the method -- you should try to include a 1-line summary) and 
the traceback from the failed test. And the traceback includes the value 
which fails.


> Ran 1 test in 0.009s
> 
> FAILED (failures=2)
> 
> Question #1:  Why is the test output NOT showing the first, passed
> test?  The one where the string is "Monty Python"?

By default, unittest shows a progress bar of dots like this:

.....F....E....FF.....S........F.......F....

That shows 37 passing tests, 1 error (the test itself is buggy), 1 
skipped test, and five failed tests. If everything passes, you just see 
the series of dots.

To see passing tests listed as well as failures, you need to pass the 
verbose flag to unittest:

python -m unittest --verbose


> Question #2:  Before I hit the docs to learn about subTest, I tried a
> bunch of different ways to make that for loop to work inside the
> class, but I guess my current poor knowledge of OOP did me in.  But I
> KNOW there has to be a way to accomplish this WITHOUT using subTest.
> After all, subTest was not introduced until Python 3.4.  So how should
> I go about this without using subTest?

*shrug* Just leave the subTest out.

The difference is:

- without subtest, the first failed test will exit the test case 
  (that is, the test_* method);
- the failed test may not print out sufficient information to 
  identify which data failed.

Only if all the data passes will the entire loop run.

Before subTest, I used to write a lot of tests like:

def test_Spam(self):
    for x in bunch_of_stuff:
        self.assertEqual(func(x), expected)

which was great so long as they all passed. But as soon as one failed, 
it was twenty kinds of trouble to work out *which* one failed. So then, 
and only then, did I break it up into separate test cases:

def test_Spam_foo(self):
    self.assertEqual(func(foo), expected)

def test_Spam_bar(self):
    self.assertEqual(func(bar), expected)

def test_Spam_baz(self):
    self.assertEqual(func(baz), expected)


subTest is a MUCH BETTER WAY to solve this problem!!!! 

/me dances the Dance Of Joy


> Question #3:  In "with self.subTest(test_string = test_string):" why
> is "test_string = test_string" necessary?  Is it that using "with"
> sets up a new local namespace within this context manager?

You need to tell subTest which variables you care about, and what their 
value is. 

Your test case might have any number of local or global variables. It 
might care about attributes of the test suite. Who knows what 
information you want to see if a test fails? So rather than using some 
sort of dirty hack to try to guess which variables are important, 
subTest requires you to tell it what you want to know.

Which could be anything you like:

    with subTest(now=time.time(), spam=spam, seed=random.seed()):
        ...


you're not limited to just your own local variables. subTest doesn't 
care what they are or what you call them -- it just collects them as 
keyword arguments, then prints them at the end if the test fails.



-- 
Steve