doctest random output?

[Chris Angelico]

On Wed, Aug 30, 2017 at 1:39 AM, Stefan Ram <ram at zedat.fu-berlin.de> wrote:
> Dennis Lee Bieber <wlfraed at ix.netcom.com> writes:
>>Testing randomness itself requires statistical tests...
>
>   A perfectly random coin /can/ yield "heads" a thousand times
>   in sequence (which is very unlikely, but possible).
>
>   This behavior should fail nearly all statistical tests for
>   randomness. Yet the generator was perfectly random.
>
>   So the tests for randomness give correct answers only with
>   a certain probability ("confidence"). Insofar the concept of
>   randomness is "fuzzy" when defined as an observable
>   property of an otherwise "black box".
>
>   The tests in the OP test only what one can test with
>   certainity, which might be reasonable.
>
>   To gain confidence in a function providing sufficiently
>   "random" results other measures might be added, such as
>   a code review (view the generator as a "white box").

The point of unit testing (of which doctests are a form) is generally
that you test THIS function, without needing to test everything else.
Testing whether random.random() is "sufficiently random" is not the
point of the doctest. For a non-trivial example, consider my dice
roller; I don't have a Python function for it, but it's a feature of
my D&D MUD. You pass it a string that details the dice you want to
roll, and it rolls them:

>>> roll d20
You roll d20: 3
>>> roll d20 + 5
You roll d20: 14
You add a bonus of 5
For d20 + 5, you total: 19
>>> roll 3d6+    d8 -2
You roll 3d6: 1, 5, 5, totalling 11.
You roll d8: 2
You add a bonus of -2
For 3d6+    d8 -2, you total: 11

This is fine as documentation. The trouble is that, for testing, we
have to logically accept any integer from 1 to 20 as "correct", and
doctest doesn't support that. I don't care, in this test, whether the
dice roller is "fair" (that it has equal probability of returning each
value) - what I care about is whether, when you enter a particular
string of dice descriptions, you get back a proper pattern of rolls.

And I don't think doctest is flexible enough to handle this without
some sort of monkeypatching - unless you code your function to use
NOTHING other than random.random(), and then you can reliably just
seed the RNG.

ChrisA