[issue22606] Inconsistency between Python 2 and PyPy regarding future imports

[Terry J. Reedy]

Terry J. Reedy added the comment:

A difference in internal behavior between CPython and PyPy is not necessarily a CPython bug, or even a PyPy bug.  It may just be an implementation choice.  So ignore PyPy and consider Python doc versus CPython behavior.

Running the three quoted lines by themselves gives SyntaxError.  compiling the ast object does the same.  So this is one of those SyntaxErrors not noticed (or at least not enforced) until the compile phase. This was specified in PEP 236 and still is in the Reference: "A future statement is recognized and treated specially at compile time".  Whenever the compiler raises SyntaxError, it declares the AST to be syntactically invalid.  How can this be?

The ast doc starts with "The ast module helps Python applications to process trees of the Python *abstract syntax grammar*. ... this module helps to find out programmatically what the current grammar looks like." (asterisks added). The abstract grammar does *not* include a 'Future' statement. In your code, x.body[1] is an ImportFrom object.  So the ast from the code above does not violate the abstract grammar and there is no parsing bug.

We should either either close this as 'not a bug' or change to a doc issue to add something about the ast abstract grammar being 'looser' than the defacto context-sensitive Python grammar.  Perhaps something like "The language defined by the context-free LL(1) abstract grammer is a superset of the actual context-sensitive Python language. (Consider position-dependent future statements.)  Consequently, some ast trees generate a SyntaxError when compiled."

It is somewhat arbitrary when a print statement is converted to a print expression -- while parsing, or while compiling.  The two implementations made different choices.

Since the parser does recognize a Future statement and change the ast objects it creates, it would be possible to add a Future object to ast trees.  But it would do no good as far as the syntax check goes.  The context-free LL(1) parser could not recognize a violation of the context-sensitive constraint.

----------
nosy: +benjamin.peterson, terry.reedy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22606>
_______________________________________