Probabilistic unit tests?

[Roy Smith]

In article <b312f3e7-5c73-486e-925e-da8343963fb6 at googlegroups.com>,
 Nick Mellor <thebalancepro at gmail.com> wrote:

> Hi,
> 
> I've got a unit test that will usually succeed but sometimes fails. An 
> occasional failure is expected and fine. It's failing all the time I want to 
> test for.
> 
> What I want to test is "on average, there are the same number of males and 
> females in a sample, give or take 2%."
> [...]
> My question is: how would you run an identical test 5 times and pass the 
> group *as a whole* if only one or two iterations passed the test? Something 
> like:
> 
>     for n in range(5):
>         # self.assertAlmostEqual(...)
>         # if test passed: break
>     else:
>         self.fail()

I would do something like:

def do_test_body():
   """Returns 1 if it passes, 0 if it fails"""

results = [do_test() for n in range(number_of_trials)]
self.assert(sum(results) > threshold)

That's the simple part.

The more complicated part is figuring out how many times to run the test 
and what an appropriate threshold is.  For that, you need to talk to a 
statistician.