[Python-Dev] Purpose of Doctests [Was: Best practices for Enum]

[Olemis Lang]

Hi !
... sorry , I could not avoid to reply this message ...

On 5/20/13, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
>
> On 20 May 2013, at 18:26, Mark Janssen <dreamingforward at gmail.com> wrote:
>
>>>> I'm hoping that core developers don't get caught-up in the "doctests are
>>>> bad
>>>> meme".
>>>>
>>>> Instead, we should be clear about their primary purpose which is to
>>>> test
>>>> the examples given in docstrings.
>>>> In other words, doctests have a perfectly legitimate use case.
>>>
>>> But more than just one ;-)  Another great use has nothing to do with
>>> docstrings:  using an entire file as "a doctest".   This encourages
>>> writing lots of text explaining what you're doing,. with snippets of
>>> code interspersed to illustrate that the code really does behave in
>>> the ways you've claimed.
>>
>> +1, very true.  I think doctest excel in almost every way above
>> UnitTests.  I don't understand the popularity of  UnitTests, except
>> perhaps for GUI testing which doctest can't handle.  I think people
>> just aren't very *imaginative* about how to create good doctests that
>> are *also* good documentation.
>>
>

With enhanced doctests solution in mind ...

> Doc tests have lots of problems for unit testing.
>
> * Every line is a test with *all* output part of the test - in unit tests
> you only assert the specific details you're interested in

custom output checkers

> * Unordered types are a pain with doctest unless you jump through hoops

( custom output checkers + doctest runner ) | (dutest __tc__ global var)

> * Tool support for editing within doctests is *generally* worse

this is true , let's do it !

> * A failure on one line doesn't halt execution, so you can get many many
> reported errors from a single failure

it should if REPORT_ONLY_FIRST_FAILURE option [1]_ is set .

> * Try adding diagnostic prints and then running your doctests!

I have ... dutest suites for my Trac plugins do so . However logging
information is outputted to /path/to/trac/env/log/trac.log ... so a
tail -f is always handy .

> * Tools support in terms of test discovery and running individual tests is
> not as smooth

dutest offers two options since years ago MultiTestLoader combines
multiple test loaders to *load* different kinds of tests at once from
a module , whereas a package loader performs test discovery . These
loader objects are composable , so if an instance of MultiTestLoader
is supplied in to the package test loader then multiple types of tests
are loaded out of modules all over across the package hierarchy .

Indeed , in +10 years of Python development I've never used
unittest(2) discovery, and even recently implemented the one that's
used in Apache™ Bloodhound test suite . Unfortunately I've had no much
time to spend on improving all this support in dutest et al.

> * Typing >>> and ... all the time is really annoying

... I have faith ... there should be something like this for vim ... I
have faith ... ;)

> * Doctests practically beg you to write your code first and then copy and
> paste terminal sessions - they're the enemy of TDD

Of course , not , all the opposite . If the approach is understood
correctly then the first thing test author will do is to write the
code «expected» to get something done . When everything is ok with API
code style then write the code . Many problems in the API and
inconsistencies are thus detected early .

> * Failure messages are not over helpful and you lose the benefit of some of
> the new asserts (and their diagnostic output) in unittest

(custom ouput checkers) | ( dutest __tc__ variable )

> * Tests with non-linear code paths (branches) are more painful to express in
> doctests
>

that's a fact , not just branches , but also exceptions

Beyond this ...

My really short answer is that I do not agree with this . Like I just
said in previous messages with enhanced support like the one offered
by dutest (i.e. __tc__ global var bound to an instance of
unittest.TestCase) it's possible to invoke each and every unittest
assertion method . So this may be seen all the other way round
«unittest machinery is already used without even declaring a single
test class» ... and so on ...

... so , in concept , there is no real benefit in using unittest over
doctest *if* doctest module is eventually upgraded .

[...]
>
> However doctests absolutely rock for testing documentation / docstring
> examples.
>

FWIW , +1

[...]

.. [1] doctest.REPORT_ONLY_FIRST_FAILURE
        (http://docs.python.org/2/library/doctest.html#doctest.REPORT_ONLY_FIRST_FAILURE)

-- 
Regards,

Olemis.

Apache™ Bloodhound contributor
http://issues.apache.org/bloodhound

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article: