Probabilistic unit tests?

[Alister]

On Thu, 10 Jan 2013 17:59:05 -0800, Nick Mellor wrote:

> Hi,
> 
> I've got a unit test that will usually succeed but sometimes fails. An
> occasional failure is expected and fine. It's failing all the time I
> want to test for.
> 
> What I want to test is "on average, there are the same number of males
> and females in a sample, give or take 2%."
> 
> Here's the unit test code:
> import unittest from collections import counter
> 
> sex_count = Counter()
> for contact in range(self.binary_check_sample_size):
>     p = get_record_as_dict()
>     sex_count[p['Sex']] += 1
> self.assertAlmostEqual(sex_count['male'],
>                        sex_count['female'], delta=sample_size * 2.0 /
>                        100.0)
> 
> My question is: how would you run an identical test 5 times and pass the
> group *as a whole* if only one or two iterations passed the test?
> Something like:
> 
>     for n in range(5):
>         # self.assertAlmostEqual(...)
>         # if test passed: break
>     else:
>         self.fail()
> 
> (except that would create 5+1 tests as written!)
> 
> Thanks for any thoughts,
> 
> Best wishes,
> 
> Nick

unit test are for testing your code, not checking if input data is in the 
correct range so unless you are writing a program intended to generate 
test data I don't see why unit test are appropriate in this case.





-- 
Under capitalism, man exploits man.  Under communism, it's just the 
opposite.
		-- J.K. Galbraith