[Python-ideas] Hooking between lexer and parser

[random832 at fastmail.us]

On Sun, Jun 7, 2015, at 01:59, Nick Coghlan wrote:
> 1. The standard Python tokeniser has already taken care of converting
> the byte stream into Unicode code points, and the code point stream
> into tokens (including replacing leading whitespace with the
> structural INDENT/DEDENT tokens)

Remember that balanced brackets are important for this INDENT/DEDENT
transformation. What should the parser do with indentation in the
presence of a hook that consumes a sequence containing unbalanced or
mixed brackets?