[Python-ideas] doctest

[Devin Jeanpierre]

On Fri, Feb 17, 2012 at 4:57 PM, Mark Janssen <dreamingforward at gmail.com> wrote:
> I find myself wanting to use doctest for some test-driven development,
> and find myself slightly frustrated and wonder if others would be
> interested in seeing the following additional functionality in
> doctest:
>
> 1. Execution context determined by outer-scope doctest defintions.

I'm not sure what you mean, but it might be relevant that Sphinx lets
you define multiple scopes for doctests. I feel like its approach is
the right one, but it isn't reusable in Python docstrings. That said,
I think users of doctest have moved away from embedded doctests in
docstrings -- it encourages doctests to have way too many "examples"
(test cases), which reduces their usefulness as documentation.

> 2. Smart Comparisons that will detect output of a non-ordered type
> (dict/set), lift and recast it and do a real comparison.

I think it's better to just always use ast.literal_eval on the output
as another form of testing for equivalence. This could break code, but
probably not any code worth caring about.

(in particular,
    >>> print 'r""'
    ""
would pass in a literal_eval-ing system, but not in some other system)

> Without #1, "literate testing" becomes awash with re-defining re-used
> variables which, generally, also detracts from exact purpose of the
> test -- this creates testdoc noise and the docs become less useful.
> Without #2, "readable docs" nicely co-aligning with "testable docs"
> tends towards divergence.
>
> Perhaps not enough developers use doctest to care, but I find it one
> of the more enjoyable ways to develop python code -- I don't have to
> remember test cases nor go through the trouble of setting up
> unittests.   AND, it encourages agile development.  Another user wrote
> a while back of even having a built-in test() method.  Wouldn't that
> really encourage agile developement?  And you wouldn't have to muddy
> up your code with "if __name__ == "__main__": import doctest, yadda
> yadda".
>
> Anyway... of course patches welcome, yes...  ;^)

Not exactly... doctest has no maintainer, and so no patches ever get
accepted. If you want to improve it, you'll have to fork it. I hope
you're that sort of person, because doctest can totally be improved.
It suffers a lot from people thinking of what it is rather than what
it could be. :(

I've in the past worked a bit on improving doctest in a fork I
started. Its primary purpose was originally to add Cram-like "shell
doctests" to doctest (see http://pypi.python.org/pypi/cram ), but
since then I started working on other bits here and there. The work
I've done is available at
https://bitbucket.org/devin.jeanpierre/doctest2 (please forgive the
presumptuous name -- I'm considering a rename to "lembas".)

The reason I've not worked on it recently is that the problems have
gotten harder and my time has run short. I would be very open to
collaboration or forking, although I also understand that a largeish
expansion with redesigned internals created by an overworked student
is probably not the greatest place to start.

This is all assuming your intentions are to contribute rather than
only suggest. Not that suggestions aren't welcome, I suppose, but
maybe not here. doctest is not actively developed or maintained
anywhere, as far as I know. (I want to say "except by me", because
that'd make me seem all special and so on, but I haven't committed a
thing in months.)

Mostly, I feel a bit like this thread could accidentally spawn
parallel / duplicated work, so I figured I'd put what I have out here.
Please don't take it for more than it is, doctest2 is still a work in
progress (and, worse, its source code is in the middle of two feature
additions!)

I definitely hope you help to make the doctest world better. I think
it fills a role that should be filled, and its neglect is unfortunate.

-- Devin