doctest exceptoin pickiness (was RE: eval(repr(x)) == x)

[Tim Peters]

[Alex Martelli]
> I still can't see it.  I generally use doctest for unit tests,
> and it does lend itself to working that way (just cutting and
> pasting an interactive session) without dependency on eval and
> repr being inverse.  The *only* p.i.t.a with doctest is that with every
> Python release the exact wording of some exception or other changes,

Also pretty common for AttributeErrors to turn into TypeErrors, and vice
versa (alas -- it's a fuzzy distinction!).

> so that all tests that involve that exception "break" until they've
> been 'fixed' to reflect the wording in the new version (and then you
> can't run them under the previous version any more) -- basically it's
> very impractical for test suites intended to survive from one Python
> version to the next, or work across two Python versions.  But that
> wouldn't be fixed by any change to repr...

Guido whines about this too, and I always tell him to grow up <wink>.  Since
doctest's initial and still-best purpose is to verify examples in
docstrings, and users (rightfully) call it "a bug" when canned examples
clash with what they actually see, doctest deliberately complains when a
canned error message gets out of synch with reality.  It would be easy to
relax that, but I wouldn't want to make tolerance the default behavior.

Ensuring just "does or does not raise *some* exception" would be the easiest
alternative to provide.  An insane extension could invent markup tagging
different messages with different versions (of Python, of the app, of the
system libraries, ...), and pick the right one to compare against at
runtime.  Better than either of those, though, is to capture
version-independent examples from the start, e.g.

>>> try:
...     myfunc(13)
... except OddNumberError:
...     pass

That communicates to the reader that OddNumberError is expected, but doesn't
gripe if the detail changes from "odd number:  go away!" to "odd number 13
supplied, even number required".  Change the code to raise PrimeNumberError,
though, and it will complain -- unless PrimeNumberError is a subclass of
OddNumberError, which would be a bug of a different sort, or a
backward-compatibility hack the very need for which a failing doctest could
point out).

A good doctest-- like good docs --requires thought.  Examples that reveal
too much about implementation details aren't just brittle for doctest's
purposes, they make for poor documentation too (how is the docstring reader
supposed to guess which parts of the example aren and aren't expected to
change over time?)!  For doctest's primary purpose, making life difficult
when fed poor examples is "a feature".