[New-bugs-announce] [issue40585] compile_command exec not raising syntax error with new PGEN Parser

[Matthias Bussonnier]

New submission from Matthias Bussonnier <bussonniermatthias at gmail.com>:

compile_command  used to produce syntax error in exec mode:

```
$ python -c "import codeop; codeop.compile_command('raise = 2', symbol='exec')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...python3.8/codeop.py", line 125, in compile_command
    return _maybe_compile(_compile, source, filename, symbol)
  File "...python3.8/codeop.py", line 100, in _maybe_compile
    raise err1
  File "...python3.8/codeop.py", line 87, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "...python3.8/codeop.py", line 105, in _compile
    return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT)
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax
```

This happens to not be the case anymore in master, where it simply return `None` for above invalid input. 

```
$ python -c "import codeop; codeop.CommandCompiler()('raise = 2', symbol='exec')"
```

or many other:

```
 $ python
Python 3.9.0a6+ (heads/master:2cc9b8486d, May 10 2020, 15:52:00)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codeop;
>>> codeop.compile_command('def a-b', symbol='exec')
>>> codeop.compile_command('await?', symbol='exec')
>>> codeop.compile_command('=!=', symbol='exec')
>>> codeop.compile_command('a await raise b', symbol='exec')
>>> codeop.compile_command('a await raise b?+1', symbol='exec')
```


It is problematic as this is used in many places to decide whether code is valid, and for example in IPython to know wether we should insert a new line, or try to execute the code. 

It seem to be due to the new PGEN parser as setting PYTHONOLDPARSER solve the issue:

```
$ PYTHONOLDPARSER=1 python -c "import codeop; codeop.CommandCompiler()('raise = 2', symbol='exec')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 171, in __call__
    return _maybe_compile(self.compiler, source, filename, symbol)
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 100, in _maybe_compile
    raise err1
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 87, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 136, in __call__
    codeob = compile(source, filename, symbol, self.flags, True)
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax
```

`single` and `eval` appear to behave fine.

----------
components: Interpreter Core
messages: 368594
nosy: mbussonn
priority: normal
severity: normal
status: open
title: compile_command exec not raising syntax error with new PGEN Parser
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40585>
_______________________________________