[Python-Dev] small Grammar questions

[Brett Cannon]

On Feb 19, 2008 1:38 PM, Andrew Dalke <dalke at dalkescientific.com> wrote:
> I'm finishing up a PLY lexer and parser for the current CVS version of
> the Python grammar.  As part of it I've been testing a lot of dark
> corners in the grammar definition and implementation.  Python 2.5 has
> some small and rare problems which I'm pleased to note have been
> pretty much fixed in Python 2.6.
>
> I have two questions about the grammar definition.  Here's a
> difference between 2.5 and 2.6.
>
> % cat x.py
> c = "This is 'c'"
> def spam((a) = c):
>   print a
> spam()
>
> % python2.5 x.py
> This is 'c'
> % python2.6 x.py
>   File "x.py", line 2
>     def spam((a) = c):
> SyntaxError: non-default argument follows default argument
>
>
> I don't understand why that's changed.  This shouldn't be a
> SyntaxError and there is no non-default argument following the default
> argument.
>

The error might be odd, but I don't see why that should be allowed
syntax. Having a parameter surrounded by a parentheses like that makes
no sense in a context of a place where arbitrary expressions are not
allowed.

> Note that this is still valid
>
> >>> def spam((a,) = c):
> ...   pass
> ...
>
> I think 2.6 is incorrect.  According to the documentation at
>   http://docs.python.org/ref/function.html
>
> defparameter    ::=     parameter ["=" expression]
> sublist ::=     parameter ("," parameter)* [","]
> parameter       ::=     identifier | "(" sublist ")"
> funcname        ::=     identifier

Looking at Grammar/Grammar, the relevant rules are::

  parameters: '(' [varargslist] ')'
  varargslist: ((fpdef ['=' test] ',')*
              ('*' NAME [',' '**' NAME] | '**' NAME) |
              fpdef ['=' test] (',' fpdef ['=' test])* [','])
  fpdef: NAME | '(' fplist ')'
  fplist: fpdef (',' fpdef)* [',']

>From what I can tell the grammar does not prevent it. But it is
possible that during AST creation or bytecode compilation a specific
check is made that is throwing the exception...

And checking Python.ast.c:672 seems to suggest it is something the AST
is doing. I don't have time to dig deeper, but if you tried ``svn
blame`` and found out when that line was added the log message give
some clues as to why this is occurring.

>
>
> Second question is about trailing commas in list comprehension.  I
> don't understand why the commented out line is not allowed.
>
> [x for x in 1]
> #[x for x in 1,]  # This isn't legal
> [x for x in 1,2]
> [x for x in 1,2,]
> [x for x in 1,2,3]
>
> The Grammar file says
>
> # Backward compatibility cruft to support:
> # [ x for x in lambda: True, lambda: False if x() ]
> # even while also allowing:
> # lambda x: 5 if x else 2
> # (But not a mix of the two)
> testlist_safe: old_test [(',' old_test)+ [',']]
>
> but if I change it to also allow
>
>     testlist_safe : old_test ','
>
> then PLY still doesn't report any ambiguities in the grammar and I
> can't find an expression that exhibits a problem.
>
> Could someone here enlighten me?
>

Are you asking why the decision was made to make the expression
illegal, or why the grammar is flagging it is wrong?

-Brett