doctest fails to NORMALIZE_WHITESPACE ?

[Tim Peters]

[David MacKay, having fun with doctest]
 ...
> I've got a follow-up question motivated by my ugly backslash continuations.

[Tim]
>> When Python doesn't "look clean", it's not Python -- and backslash
>> continuation & semicolons often look like dirt to the experienced
>> Python's eye.

> The reason I was making those ugly single-line monsters was I had somehow got
> the impression that each fresh line starting ">>>" was tested
> _completely_separately_ by doctest; so, to preserve state I thought I had to write
>
> >>> a=[2, 5, 1];  a.sort();  print a
> [1, 2, 5]
>
> rather than
>
> >>> a=[2, 5, 1]
> >>> a.sort()
> >>> print a
> [1, 2, 5]
>
> But I see now I was wrong.

Yup, that would be pretty unusable ;-)

> Question:
> Does doctest always preserve state throughout the entire sequence of tests
>     enclosed by `"""`, as if the tests were entered in a single interactive session?

Pretty much.  doctest can take inputs from several places, like (as
you're doing) docstrings, but also from files, or from the values in a
__test__ dictionary.  I'll note parenthetically that doctests are
heavily used in Zope3 and ZODB development, and "tutorial doctests" in
files have proved to be pleasant & very effective.  For example,

<http://svn.zope.org/ZODB/trunk/src/ZODB/tests/multidb.txt?rev=39908&view=markup>

That entire file is "a doctest", and contains as much expository prose
as code.  It's written in ReST format, and tools in Zope3 can present
such doctest files as nicely formatted documentation too.  Unlike most
docs, though, each time we run ZODB's or Zope3's test suite, we
automatically verify that the examples in _this_ documentation are
100% accurate.  Standard practice now is to write a tutorial doctest
for a new feature first, before writing any implementation code; this
folds in some aspects of test-driven development, but  with some care
leaves you with _readable_ testing code and a tutorial intro to the
feature too.  That doctest makes it a little easier to write prose
than to write code is a bias that's amazingly effective in getting
people to write down what they _think_ they're doing ;-)

Anyway, running any piece of Python code requires a global namespace,
and doctest uses a single (but mutable!) global namespace for each
_batch_ of tests it runs.  In the case of a doctest file, a single
global namespace is used across the entire file.  In your case,
letting doctest extract tests from module docstrings, one global
namespace is created per docstring.  In that case, the globals the
tests in a docstring use are initially a shallow copy of the module's
__dict__.  That way the tests can "see" all the top-level functions
and classes and imports (etc) defined in the module, but can't mutate
the module's __dict__ directly.  Assignments within a doctest alter
bindings in the same namespace object, so these bindings are also
visible to subsequent code in the same docstring.  That's a
long-winded way of expanding on section 5.2.3.3 ("What's the Execution
Context?") in the docs.

> Or is there a way to instruct doctest to forget its state, and start
>  the next `>>>` with a clean slate?

Goodness no -- and nobody would want that.  You would lose _all_
globals then, including losing the ability to refer to functions and
classes (etc) defined in the module.

If, for some reason, you want to destroy some particular binding
within a doctest, then you do that the same way you destroy a global
binding outside of doctest, with `del`; e.g.,

"""
>>> x = range(1000000)
>>> len(x)
1000000
>>> del x  # free the memory for the giant list
>>> x
Traceback (most recent call last):
   ...
NameError: name 'x' is not defined
"""

works fine as a doctest.