Sphinx Doctest: test the code without comparing the output.

[Steven D'Aprano]

On Sat, 21 Sep 2013 09:25:26 -0700, Luca Cerone wrote:

>> And if you ignore the output, the error won't be caught either. What's
>> the difference?
>> 
>> >>> 1 + 1  #doctest:+IGNORE_OUTPUT  (not a real directive)
>> 1000
>> 
>> 
> The difference is that in that case you want to check whether the result
> is correct or not, because you expect a certain result.
> 
> In my case, I don't know what the output is, nor care for the purpose of
> the tutorial. What I care is being sure that the command in the tutorial
> is correct, and up to date with the code.
> 
> If you try the following, the test will fail (because there is a typo in
> the code)
> 
> .. doctest:: example
> 
>    >>> printt "hello, world"
> 
> and not because the output doesn't match what you expected.


That is not how doctest works. That test fails because its output is:

SyntaxError: invalid syntax


not because doctest recognises the syntax error as a failure. Exceptions, 
whether they are syntax errors or some other exception, can count as 
doctest *successes* rather than failures. Both of these count as passing 
doctests:


>>> x++  # Python is not C!
Traceback (most recent call last):
  ..
SyntaxError: unexpected EOF while parsing

>>> 1/0
Traceback (most recent call last):
  ..
ZeroDivisionError: division by zero


Consequently, doctest only recognises failures by their output, 
regardless of whether that output is a value or an exception.


> Even if the command is correct:
> 
> .. doctest:: example_2
> 
>    >>> print "hello, world"
> 
> this text will fail because doctest expects an output. I want to be able
> to test that the syntax is correct, the command can be run, and ignore
> whatever the output is.

The only wild-card output that doctest recognises is ellipsis, and like 
all wild-cards, can match too much if you aren't careful. If ellipsis is 
not matching exactly what you want, add some scaffolding to ensure a 
match:

   >>> import random
   >>> x = random.uniform(0,100)
   >>> print "x =", x  # doctest:+ELLIPSIS
   x = ...


will work. But a better solution, I think, would be to pick a 
deterministic result:

    >>> import random
    >>> random.seed(100)
    >>> random.uniform(0, 100)
    14.566925510413032


Alas, that only works reliably if you stick to a single Python version. 
Although the results of calling random.random are guaranteed to be stable 
across versions, random functions build on top of random.random like 
uniform are not and may need to be protected with a version check.



-- 
Steven