doctest random output?

[Steve D'Aprano]

On Mon, 28 Aug 2017 07:41 pm, Leam Hall wrote:

> Is this a good way to test if random numeric output? It seems to work
> under Python 2.6 and 3.6 but that doesn't make it 'good'.

That depends on what you are actually testing. If you are intending to test the
statistical properties of random, google for the Die Hard tests and start by
porting them to Python.

But if you're just hoping to test your library's APIs, that's trickier than it
seems. Unfortunately, Python doesn't guarantee that the exact output of the
random module is stable across bug fix releases, except for random.random
itself. So this is safe:

def my_thing():
    """blah blah blah

    >>> random.seed(45)
    >>> my_thing()  # calls random.random
    0.738270225794931

    """

But this is not:

def my_thing():
    """blah blah blah

    >>> random.seed(45)
    >>> my_thing()  # calls random.int
    4

    """


That makes doctesting anything related to random a PITA. Here are some
suggestions, none of them are really great:


(1) Disable doctesting for that example, and treat it as just documentation:

def my_thing():
    """blah blah blah

    >>> my_thing()  #doctest:+SKIP
    4

    """


(2) Monkey-patch the random module for testing. This is probably the worst idea
ever, but it's an idea :-)


def my_thing():
    """blah blah blah

    >>> import random
    >>> save = random.randint 
    >>> try:
    ...     random.randint = lambda a, b: 4  
    ...     my_thing()
    ... finally:
    ...     random.randint = save
    4

    """

That makes for a fragile test and poor documentation.


(3) Write your functions to take an optional source of randomness, and then in
your doctests set them:

def my_thing(randint=None):
    """blah blah blah

    >>> my_thing(randint=lambda a,b: 4)
    4

    """
    if randint is None: 
        from random import randint
    ...



(4) Write your doctests to test the most general properties of the returned
results:


def my_thing(randint=None):
    """blah blah blah

    >>> num = my_thing()
    >>> isinstance(num, int) and 0 <= my_thing() <= 6
    True

    """




-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.