[Python-ideas] relaxing keyword usage restrictions

[Terry Reedy]

On 9/9/2011 2:04 AM, H Krishnan wrote:
> Hi,
> I have read through the comments and the (largely negative) votes for
> this idea.
>
> The main argument against it seems to be readability. I feel the
> examples I chose were distracting. The intent of the examples was to
> show the semantics and not to suggest that anyone should write such
> code.

The parser has to be ready for and work correctly with the worst horror 
anyone *could* write.
   if in in in or and and and:

One can write totally unreadable code even with the current syntax:
> if not was is can:
>     return take

Except that *is* readable *because* keywords are keywords and 
identifiers are identifiers. 'if not x is y' implies that x and y should 
be boolean values and we want them to have opposite values to proceed.

> One cannot however write readable code such as the following:
> if yield > principal:
>     return = yield - principal

Funny you should choose that example. With a slight change
   myreturn = yield - principal
it is legal syntax today with 'yield' interpreted as a keyword. So it 
cannot be interpreted as an identifier without making Python grammar 
ambiguous and unparseable with its current parser.

Because C grammar allows typedef names, which are just identifiers, to 
be used as pseudo-keywords, without being distinguished by a real 
keyword or other mechanism, it is ambiguous. The ambiguity is resolved 
by making C globally context sensitive. Therefore, C cannot be parsed 
and compiled with a LR or LL compiler, such as produced by YACC. The 
parser has to be escaped to resolve the ambiguity and everyone who 
writes C parsers has to patch the output of any automatic parser generator.

Having a sequence of characters be ambiguously either a keyword or an 
identifier is a bad idea. They are really different things. They have 
different grammatical functions.

If one wants everything that looks like an identifier by be usable as an 
identifier, then there should be no identifier-like keywords. Require 
that all keystrings not be legal identifiers. Make them consist of or 
include some non-identifier character.

Guido could had decided that all keystrings should start or end with a 
symbol such as '$' or '`', : $if, $yield, etc. Your problem would have 
been solved. And most everyone would have a problem with harder code entry.

> The second argument was about the domino effect of such a change on
> parsing and syntax highlighting tools. That is true for any change to
> the core language, though the proposed change might cause more impact
> than most others.

Hardly in the same ballpark, I think. 3.2 had no core language changes. 
3.3 might add 'yield from'. Syntax highlighters should not be affected. 
Adding a new keyword should also have no impact as any such tool should 
get the current keywords from keyword.kwlist.

> If syntax highlighting tools can use python's ast and
> other modules to identify keywords (instead of using their own parsers),
> the domino effect will be lesser.

Right now, it is a simple lookup.

> The use-case I had in mind was related to a customization tool we are
> developing. We want to allow users to define expressions and assign them
> to names. We 'eval' the expressions subsequently. We have to currently
> prevent users from using any python keyword as a name, which I felt was
> an unnecessary restriction.

For the present, you can remove the restriction on keywords that cannot 
appear in expressions by mechanically transforming them.

-- 
Terry Jan Reedy