Probabilistic unit tests?

[Nick Mellor]

Hi,

I've got a unit test that will usually succeed but sometimes fails. An occasional failure is expected and fine. It's failing all the time I want to test for.

What I want to test is "on average, there are the same number of males and females in a sample, give or take 2%."

Here's the unit test code:
import unittest
from collections import counter

sex_count = Counter()
for contact in range(self.binary_check_sample_size):
    p = get_record_as_dict()
    sex_count[p['Sex']] += 1
self.assertAlmostEqual(sex_count['male'],
                       sex_count['female'],
                       delta=sample_size * 2.0 / 100.0)

My question is: how would you run an identical test 5 times and pass the group *as a whole* if only one or two iterations passed the test? Something like:

    for n in range(5):
        # self.assertAlmostEqual(...)
        # if test passed: break
    else:
        self.fail()

(except that would create 5+1 tests as written!)

Thanks for any thoughts,

Best wishes,

Nick