unit test strategy

[Steven D'Aprano]

On Sun, 16 Sep 2012 11:38:15 -0700, Aaron Brady wrote:


> Here is an example of some repetitive code.
> 
> for view_meth in [ dict.items, dict.keys, dict.values ]:
> 	dict0= dict( ( k, None ) for k in range( 10 ) ) 
>       iter0= iter( view_meth( dict0 ) )
> 	dict.__setitem__( dict0, 0, 1 )
> 	next( iter0 )
> 	dict.__setitem__( dict0, 10, 1 )
> 	self.assertRaises( IterationError, next, iter0 )
[...]

First off, if you have any wish for this to be accepted into the standard 
library, I suggest you stick to PEP 8. Spaces on *both* sides of equals 
signs, not just one(!!!), and *no* spaces around the parenthesised 
arguments.

Secondly, this is test code. A bit of repetition is not to be concerned 
about, clarity is far more important than "Don't Repeat Yourself". The 
aim isn't to write the fastest, or most compact code, but to have good 
test coverage with tests which are *obviously* correct (rather than test 
code which has no obvious bugs, which is very different). If a test 
fails, you should be able to identify quickly what failed without running 
a debugger to identify what part of the code failed.

Thirdly, why are you writing dict.__setitem__( dict0, 0, 1 ) instead of 
dict0[0] = 1 ?


[...]
> Specifically my questions are, is the code condensed beyond legibility? 

Yes.


> Should 'chain' execute the test directly, or act as a metaprogram and
> output the test code into a 2nd file, or both?

None of the above.


> Should 'chain' create
> the iterators in a dictionary, or in the function local variables
> directly with 'exec'?

Heavens to Murgatroyd, you can't be serious.


Here's my attempt at this. Note the features:

- meaningful names (if a bit long, but that's a hazard of tests)
- distinct methods for each distinct test case
- comments explaining what the test code does
- use of subclassing


# Untested
class TestDictIteratorModificationDetection(unittest.TestCase):
    """Test that iterators to dicts will correctly detect when the
    dict has been modified, and raise an exception.
    """

    def create_dict(self):
        return dict.fromkeys(range(10))

    def testIteratorAllowed(self):
        # Iterators are allowed to run if all modifications occur
        # after the iterator is created but before it starts running.
        d = self.create_dict()
        for view in (dict.items, dict.keys, dict.values):
            # Create an iterator before modifying the dict, but do not
            # start iterating over it.
            it = iter(view(d))
            # Be careful of the order of modifications here.
            del d[2]
            d.pop(4)
            d.popitem()
            d.clear()
            # Make sure we have something to iterate over.
            d[1] = 1
            d.update({5: 1})
            d.setdefault(8, 1)
            assert d  # d is non-empty.
            next(it); next(it); next(it)
            self.assertRaises(StopIteration, next, it)
            

    def iterator_fails_after_modification(self, method, *args):
        """Iterators should not be able to continue running after
        the dict is modified. This helper method factors out the common
        code of creating a dict, an iterator to that dict, modifying the
        dict, and testing that further iteration fails.

        Pass an unbound dict method which modifies the dict in place, and
        and arguments to that method required.
        """
        d = self.create_dict()
        for view in (dict.items, dict.keys, dict.values):
            it = iter(view(d))
            next(it)  # no exception expected here
            method(d, *args)
            self.assertRaises(IterationError, next, it)

    def testIteratorFailsAfterSet(self):
        self.iterator_fails_after_modification(dict.__setitem__, 1, 1)

    def testIteratorFailsAfterDel(self):
        self.iterator_fails_after_modification(dict.__delitem__, 1)

    def testIteratorFailsAfterUpdate(self):
        self.iterator_fails_after_modification(dict.update, {5: 1})

    def testIteratorFailsAfterPop(self):
        self.iterator_fails_after_modification(dict.pop, 4)

    def testStartedIteratorFailsAfterPopItem(self):
        self.iterator_fails_after_modification(dict.popitem)

    def testStartedIteratorFailsAfterClear(self):
        self.iterator_fails_after_modification(dict.clear)

    def testStartedIteratorFailsAfterSetDefault(self):
        self.iterator_fails_after_modification(dict.setdefault, 99, 1)



class TestDictSubclassIteratorModificationDetection(
               TestDictIteratorModificationDetection):
    def create_dict(self):
        class MyDict(dict):
            pass
        return MyDict.fromkeys(range(10))


I think I've got all the methods which can mutate a dictionary. If I 
missed any, it's easy enough to add a new test method to the class.

You are free to use the above code for your own purposes, with any 
modifications you like, with two requests:

- I would appreciate a comment in the test file acknowledging my 
contribution;

- I would like to be notified if you submit this to the bug tracker.

Thanks again for tackling this project.


-- 
Steven