[New-bugs-announce] [issue15606] re.VERBOSE doesn't ignore certain whitespace

[Steven Collins]

New submission from Steven Collins:

Given the way the documentation is written for re.VERBOSE - "Whitespace within the pattern is ignored, except when in a character class or preceded by an unescaped backslash" - I would expect all three of the findall() commands below to return successfully with the same result:

Python 3.2.3 (default, Jun  8 2012, 05:37:15) 
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.findall('(?x) (?: a | b ) + ', 'abaabc')
['abaab']
>>> re.findall('(?x) (? : a | b ) + ', 'abaabc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/re.py", line 193, in findall
    return _compile(pattern, flags).findall(string)
  File "/usr/lib/python3.2/re.py", line 255, in _compile
    return _compile_typed(type(pattern), pattern, flags)
  File "/usr/lib/python3.2/functools.py", line 184, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/lib/python3.2/re.py", line 267, in _compile_typed
    return sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.2/sre_compile.py", line 491, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.2/sre_parse.py", line 692, in parse
    p = _parse_sub(source, pattern, 0)
  File "/usr/lib/python3.2/sre_parse.py", line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File "/usr/lib/python3.2/sre_parse.py", line 627, in _parse
    raise error("unexpected end of pattern")
sre_constants.error: unexpected end of pattern
>>> re.findall('(?x) ( ?: a | b ) + ', 'abaabc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/re.py", line 193, in findall
    return _compile(pattern, flags).findall(string)
  File "/usr/lib/python3.2/re.py", line 255, in _compile
    return _compile_typed(type(pattern), pattern, flags)
  File "/usr/lib/python3.2/functools.py", line 184, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/lib/python3.2/re.py", line 267, in _compile_typed
    return sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.2/sre_compile.py", line 491, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.2/sre_parse.py", line 692, in parse
    p = _parse_sub(source, pattern, 0)
  File "/usr/lib/python3.2/sre_parse.py", line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File "/usr/lib/python3.2/sre_parse.py", line 640, in _parse
    p = _parse_sub(source, state)
  File "/usr/lib/python3.2/sre_parse.py", line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File "/usr/lib/python3.2/sre_parse.py", line 520, in _parse
    raise error("nothing to repeat")
sre_constants.error: nothing to repeat
>>> 

The behavior is the same in Python 2.7. Apparently the scan for the special '(?' character sequences happens before the whitespace is stripped out. In my opinion, the behavior should be changed, the documentation should be more clear about the current behavior, or at least the errors given should be more informative (I spent an hour or two debugging the "nothing to repeat" error in my work yesterday.) Thank you.

----------
components: Regular Expressions
messages: 167803
nosy: ezio.melotti, mrabarnett, stevencollins
priority: normal
severity: normal
status: open
title: re.VERBOSE doesn't ignore certain whitespace
type: behavior
versions: Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue15606>
_______________________________________