[New-bugs-announce] [issue7089] shlex behaves unexpected if newlines are not whitespace

[Jan David Mol]

New submission from Jan David Mol <jjdmol at gmail.com>:

The shlex module does not function as expected in the presence of
comments when newlines are not whitespace. An example (attached):

>>> from shlex import shlex
>>> 
>>> lexer = shlex("a \n b")
>>> print ",".join(lexer)
a,b
>>> 
>>> lexer = shlex("a # comment \n b")
>>> print ",".join(lexer)
a,b
>>> 
>>> lexer = shlex("a \n b")
>>> lexer.whitespace=" "
>>> print ",".join(lexer)
a,
,b
>>> 
>>> lexer = shlex("a # comment \n b")
>>> lexer.whitespace=" "
>>> print ",".join(lexer)
a,b

Now where did my newline go? The comment ate it! Even though the docs
seem to indicate the newline is not part of the comment itself:

shlex.commenters:
    The string of characters that are recognized as comment beginners.
All characters from the comment beginner to end of line are ignored.
Includes just '#' by default.

----------
files: lexertest.py
messages: 93776
nosy: jjdmol2
severity: normal
status: open
title: shlex behaves unexpected if newlines are not whitespace
type: behavior
versions: Python 2.4, Python 2.5, Python 2.6, Python 2.7, Python 3.0, Python 3.1, Python 3.2
Added file: http://bugs.python.org/file15087/lexertest.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7089>
_______________________________________