[New-bugs-announce] [issue21642] "_ if 1else _" does not compile

[Joshua Landau]

New submission from Joshua Landau:

By the docs,

    Except at the beginning of a logical line or in
    string literals, the whitespace characters space,
    tab and formfeed can be used interchangeably to
    separate tokens. Whitespace is needed between two
    tokens only if their concatenation could otherwise
    be interpreted as a different token
    (e.g., ab is one token, but a b is two tokens).

"_ if 1else _" should compile equivalently to "_ if 1 else _".

The tokenize module does this correctly:

    import io
    import tokenize

    def print_tokens(string):
        tokens = tokenize.tokenize(io.BytesIO(string.encode("utf8")).readline)    

        for token in tokens:
            print("{:12}{}".format(tokenize.tok_name[token.type], token.string))

    print_tokens("_ if 1else _")
    #>>> ENCODING    utf-8
    #>>> NAME        _
    #>>> NAME        if
    #>>> NUMBER      1
    #>>> NAME        else
    #>>> NAME        _
    #>>> ENDMARKER   

but it fails when compiled with, say, "compile", "eval" or "ast.parse".

    import ast

    compile("_ if 1else _", "", "eval")
    #>>> Traceback (most recent call last):
    #>>>   File "", line 32, in <module>
    #>>>   File "<string>", line 1
    #>>>     _ if 1else _
    #>>>           ^
    #>>> SyntaxError: invalid token

    eval("_ if 1else _")
    #>>> Traceback (most recent call last):
    #>>>   File "", line 40, in <module>
    #>>>   File "<string>", line 1
    #>>>     _ if 1else _
    #>>>           ^
    #>>> SyntaxError: invalid token

    ast.parse("_ if 1else _")
    #>>> Traceback (most recent call last):
    #>>>   File "", line 48, in <module>
    #>>>   File "/usr/lib/python3.4/ast.py", line 35, in parse
    #>>>     return compile(source, filename, mode, PyCF_ONLY_AST)
    #>>>   File "<unknown>", line 1
    #>>>     _ if 1else _
    #>>>           ^
    #>>> SyntaxError: invalid token

Further, some other forms work:

    1 if 0b1else 0
    #>>> 1

    1 if 1jelse 0
    #>>> 1

See

    http://stackoverflow.com/questions/23998026/why-isnt-this-a-syntax-error-in-python

particularly,

    http://stackoverflow.com/a/23998128/1763356

for details.

----------
messages: 219614
nosy: Joshua.Landau
priority: normal
severity: normal
status: open
title: "_ if 1else _" does not compile
type: compile error
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21642>
_______________________________________