doctest random output?

[Steven D'Aprano]

On Tue, 29 Aug 2017 12:25:45 +1000, Chris Angelico wrote:

> On Tue, Aug 29, 2017 at 11:53 AM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
>> (1) Disable doctesting for that example, and treat it as just
>> documentation:
>>
>> def my_thing():
>>     """blah blah blah
>>
>>     >>> my_thing()  #doctest:+SKIP
>>     4
>>
>>     """
> 
> For a lot of functions, this completely destroys the value of
> doctesting.


"The" value? Doc tests have two values: documentation (as examples of 
use) and as tests. Disabling the test aspect leaves the value as 
documentation untouched, and arguably is the least-worst result. You can 
always write a unit test suite to perform more detailed, complicated 
tests. Doc tests are rarely exhaustive, so you need unit tests as well.



>> (2) Monkey-patch the random module for testing. This is probably the
>> worst idea ever, but it's an idea :-)
>>
>> That makes for a fragile test and poor documentation.
> 
> This highlights the inherent weakness of doctests. For proper unit
> testing, I would definitely recommend this. Maybe a hybrid of 1 and 2
> could be organized... hmm.

Doc tests should be seen as *documentation first* and tests second. The 
main roll of the tests is to prove that the documented examples still do 
what you say they do.

It makes for a horrible and uninformative help() experience to have 
detailed, complex, exhaustive doc tests exercising every little corner 
case of your function. That should go in your unit tests.

Possibly relevant: the unittest module has functionality to automatically 
extract and run your library's doctests, treating them as unit tests. So 
you can already do both.



>> (3) Write your functions to take an optional source of randomness, and
>> then in your doctests set them:
>>
>> def my_thing(randint=None):
>>     """blah blah blah
>>
>>     >>> my_thing(randint=lambda a,b: 4)
>>     4
>>
>>     """
>>     if randint is None:
>>         from random import randint
>>     ...
> 
> Unless that would be useful for other reasons, not something I like
> doing. Having code in your core that exists solely (or even primarily)
> to make testing easier seems like doing things backwards.

I see your point, and I don't completely disagree. I'm on the fence about 
this one. But testing is important, and we often write code to make 
testing easier, e.g. pulling out a complex but short bit of code into its 
own function so we can test it, using dependency injection, etc. Why 
shouldn't we add hooks to enable testing? Not every function needs such a 
hook, but some do.

See, for example, "Enemies of Test Driven Development":

https://jasonmbaker.wordpress.com/2009/01/08/enemies-of-test-driven-
development-part-i-encapsulation/


In Python, we have the best of both worlds: we can flag a method as 
private, and *still* test it! So in a sense, Python's very design has 
been created specifically to allow testing.


For a dissenting view, "Are Private Methods a Code Smell?":

http://carlosschults.net/en/are-private-methods-a-code-smell/



>> (4) Write your doctests to test the most general properties of the
>> returned results:
>>
>>
>> def my_thing(randint=None):
>>     """blah blah blah
>>
>>     >>> num = my_thing()
>>     >>> isinstance(num, int) and 0 <= my_thing() <= 6
>>     True
>>
>>     """
> 
> This is what I'd probably do, tbh.

Sometimes that's sufficient. Sometimes its not. It depends on the 
function.

For example, imagine a function that returns a randomly selected prime 
number. The larger the prime, the less likely it is to be selected, but 
there's no upper limit. So you write:

   >>> num = my_thing()
   >>> isinstance(num, int) and 2 <= num
   True


Not very informative as documentation, and a lousy test too.


> None of the options really appeal though. Personally, I'd probably
> either go with #4, or maybe something like this:
> 
> def roll(sequence):
>     """Roll a set of dice
> 
>     >>> from test_mymodule import * # ensure stable RNG 
>     >>> roll("d12 + 2d6 + 3")
>     You roll d12: 8 You roll 2d6: 1, 6, totalling 7.
>     You add a bonus of 3 For d12 + 2d6 + 3, you total: 18 
>     """
> 
> and bury all the monkey-patching into test_mymodule. 


Wait... are you saying that importing test_mymodule monkey-patches the 
current library? And doesn't un-patch it afterwards? That's horrible.

Or are you saying that test_module has its own version of roll(), and so 
you're using *that* version instead of the one in the library?

That's horrible too.

I think that once you are talking about monkey-patching things in order 
to test them, you should give up on doc tests and use unittest instead. 
At least then you get nice setUp and tearDown methods that you can use.


> It can have its own
> implementations of randint and whatever else you use. That way, at least
> there's only one line that does the messing around. I still don't like
> it though - so quite honestly, I'm most likely to go the route of "don't
> actually use doctests".

Are you saying don't use doctests for *this* problem, or don't use them 
*at all*?




-- 
Steven D'Aprano
“You are deluded if you think software engineers who can't write 
operating systems or applications without security holes, can write 
virtualization layers without security holes.” —Theo de Raadt