[Python-ideas] Hooking between lexer and parser

[Nick Coghlan]

On 8 June 2015 at 12:37, Neil Girdhar <mistersheik at gmail.com> wrote:
> Ultimately, my problem with "token transformers" is, if I'm understanding
> correctly, that we want to change Python so that not only will 3.5 have
> Token transformers, but every Python after that has to support this.  This
> risks constraining the development of the elegant solution.  And for what
> major reason do we even need token transformers so soon?  For a toy example
> on python ideas about automatic Decimal instances?  Why can't a user define
> a one character function "d(x)" to do the conversion everywhere?  I prefer
> to push for the better design even if it means waiting a year.

Neil, you're the only one proposing major structural changes to the
code generation pipeline, and nobody at all is proposing anything for
Python 3.5 (the feature freeze deadline for that has already passed).

Andrew is essentially only proposing relatively minor tweaks to the
API of the existing tokenizer module to make it more iterable based
and less file based (while still preserving the file based APIs).
Eugene Toder's and Dave Malcolm's patches from a few years ago make
the existing AST -> bytecode section of the toolchain easier to modify
and experiment with (and are ideas worth exploring for 3.6 if anyone
is willing and able to invest the time to bring them back up to date).

However, if you're specifically wanting to work on an "ideal parser
API", then the reference interpreter for a 24 year old established
language *isn't* the place to do it - the compromises necessitated by
the need to align with an extensive existing ecosystem will actively
work against your goal for a clean, minimalist structure. That's thus
something better developed *independently* of CPython, and then
potentially considered at some point in the future when it's better
established what concrete benefits it would offer over the status quo
for both the core development team and Python's end users.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia