[Python-ideas] Hooking between lexer and parser

[random832 at fastmail.us]

On Fri, Jun 5, 2015, at 22:21, Neil Girdhar wrote:
> Back in the day, I remember Lex and Yacc, then came Flex and Bison, and
> then ANTLR, which unified lexing and parsing under one common language. 
> In
> general, I like the idea of putting everything together.  I think that
> because of Python's separation of lexing and parsing, it accepts weird
> text
> like "(1if 0else 2)", which is crazy.

I don't think this really has anything to do with separation of lexing
and parsing. C rejects this (where "this" is "integer followed by
arbitrary alphabetic token") purely due to the lexing stage
(specifically, 1if or 0else would be a single "preprocessor number"
token, with no valid meaning. Of course, this has its own quirks, for
example 0xE+1 is invalid in C.)