[Python-bugs-list] [ python-Bugs-732299 ] Doctest chokes on '\x00'

[SourceForge.net]

Bugs item #732299, was opened at 2003-05-04 15:23
Message generated for change (Comment added) made by francisga
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=732299&group_id=5470

Category: Python Library
Group: Not a Bug
Status: Closed
Resolution: Invalid
Priority: 5
Submitted By: Francis Avila (francisga)
Assigned to: Tim Peters (tim_one)
Summary: Doctest chokes on '\x00'

Initial Comment:
$ cat ./dtbug.py
"""Demonstrate a python doctest bug."""

def main():
        """See here, I can't include any string literals with 
nulls in them:

        >>> x = '\x00'
        >>>

        """
        pass

if __name__ == '__main__':
        import doctest, dtbug
        doctest.testmod(dtbug)

$ python ./dtbug.py
*******************************************************
**********
Failure in example: x = ''
from line #2 of dtbug.main
Exception raised:
Traceback (most recent call last):
  File "/usr/lib/python2.2/doctest.py", line 499, in 
_run_examples_inner
    exec compile(source, "<string>", "single") in globs
TypeError: compile() argument 1 must be string 
without null bytes, not string
*******************************************************
**********
1 items had failures:
   1 of   1 in dtbug.main
***Test Failed*** 1 failures.


I'm not actually certain this is really a bug, but it 
certainly makes it impossible to demonstrate the use 
of functions that operate on binary data.


----------------------------------------------------------------------

>Comment By: Francis Avila (francisga)
Date: 2003-05-04 23:49

Message:
Logged In: YES 
user_id=771433

The docs mention a few "gotcha"s, one of which is that using 
explicit line continuations must be escaped. I nodded, but I 
don't think I understood the ramifications.

But perhaps the docs should explicitly mention the case of 
string literals intended to represent binary data?  Because 
it's not simply a case of adding another level of escapes, 
but that a null is not allowed in a string passed to compile.

For example, this works fine:

>>> x = '\x01'
>>> 

(This is why I was confused, because in my docstring I had 
other examples which used *non-null* binary data passed 
to functions, and these examples passed doctest.)

Another workaround, which doesn't involve making 
docstring text not match interpreter text, is to use chr(0):

>>> x = chr(0)
>>> 

This will only be useful in very specific cases, though, like 
this artificial example.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-05-04 23:07

Message:
Logged In: YES 
user_id=31435

This isn't a bug, can't be helped, but can be avoided.  You 
have to be aware that the doctest goes through *two* 
stages of string processing:  the Python interpreter does 
one when it's reading your source code.  At that time, the 
\x00 in your doctest string gets changed into a NUL byte.  
Similarly, all other backslash escape sequences are also 
processed.

The *result* of that is the *input* string doctest sees, and 
doctest has to pass its input on to the compiler again, in 
order to run your tests.

The easiest way to avoid the problem is to make the 
docstring a raw string (stick the letter 'r' -- without the 
quotes -- at the front of the docstring).  Another way is to 
double-up your backslashes, as the docs point out.  When 
Python sees

"\x00"

it produces a four-character string, a single backslash 
followed by "x00".  That's the one you want doctest to pass 
on to the compiler.  The raw string

r"\x00"

gets to the same end via a different mechanism.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=732299&group_id=5470