Probabilistic unit tests?

[Roy Smith]

In article 
<693d4bb1-8e1e-4de0-9d4d-8a136ea70ef4 at pp8g2000pbb.googlegroups.com>,
 alex23 <wuwei23 at gmail.com> wrote:

> On 11 Jan, 13:34, Steven D'Aprano <steve
> +comp.lang.pyt... at pearwood.info> wrote:
> > Well, that's not really a task for unit testing. Unit tests, like most
> > tests, are well suited to deterministic tests, but not really to
> > probabilistic testing. As far as I know, there aren't really any good
> > frameworks for probabilistic testing, so you're stuck with inventing your
> > own. (Possibly on top of unittest.)
> 
> One approach I've had success with is providing a seed to the RNG, so
> that the random results are deterministic.

Sometimes, a hybrid approach is best.

I was once working on some code which had timing-dependent behavior.  
The input space was so large, there was no way to exhaustively test all 
conditions.  What we did was use a PRNG to drive the test scenarios, 
seeded with the time.  We would print out the seed at the beginning of 
the test.  This let us explore a much larger range of the input space 
than we could have with hand-written test scenarios.

There was also a mode where you could supply your own PRNG seed.  So, 
the typical deal would be to wait for a failure during normal (nightly 
build) testing, then grab the seed from the test logs and use that to 
replicate the behavior for further study.