[Python-ideas] Hooking between lexer and parser

[random832 at fastmail.us]

On Sun, Jun 7, 2015, at 09:24, Andrew Barnert wrote:
> I'm pretty sure that just doing nothing special here means you get a
> SyntaxError from the parser. Although I probably need more test cases.

I'm actually talking about what happens if the _untransformed_ stream
contains an unbalanced bracket that the hook is supposed to eliminate
(and/or supply the balancing one). My mental model of this idea was that
the "lexer" generates the entire untransformed (but including
indent/dedent magic etc) token sequence, then supplies it to the hook.