[Python-ideas] AST Transformation Hooks for Domain Specific Languages

[Nick Coghlan]

On Fri, Apr 8, 2011 at 11:15 PM, Eugene Toder <eltoder at gmail.com> wrote:
> So registering dsl will be client's responsibility, rather than
> something module can do for itself.

Yep, but if you're at the point of using a DSL, that's likely to be
part of a larger framework which can take care of registering the DSL
for you before you import any modules that need it.

That answer applies whether this is a standard language feature or
part of an import hook that uses its own custom compiler.

> If this is OK, we can achieve similar effect without any changes to
> Python -- for example, with import hooks. One can write a hook that
> applies whatever AST transformations to modules loaded from specific
> locations.

Yeah, the big downside is having to almost completely reimplement the
AST compiler in order to do it that way (since the built-in one would
choke on the syntax extension). That isn't *hard* so much as it is
tedious (especially compared to tweaking the existing one in place on
a clone of the main repo).

Once the AST has been transformed, of course, the existing compiler
could still be used.

Note something I haven't looked into yet is whether or not the CPython
parser can even generate a different node type for this, or manage the
"automatic stringification" of the DSL body. The former issue wouldn't
be too hard to handle (just add a "syntax" attribute to the existing
Function node, with "node.syntax is None" indicating standard Python
code, but I'm not sure about the second one (although worst case would
be to require use of the docstring to cover anything that didn't fit
with standard Python syntax - arguably not a bad idea anyway, since it
would be a lot friendlier to non-DSL aware Python tools).

> We can make AST transformation a part of module itself, e.g. with some
> kind of "eager decorator". Taking your example:
>
> @@dsl.sql.query
> def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql:
>   select address
>   from people
>   where name = {name} and dob = {dob}
>
> Eager decorator has to be used by the fully qualified name. Parser
> will import (and execute) defining module (dsl.sql in this example)
> while compiling a module that uses it (not when module is executed, as
> with normal decorator).

But what would the eager decorator buy you over just specifying a
different dialect in the "from" clause?

I'm not sure I'll ever actually create a prototype of this (lots of
other things on the to-do list), but I found the idea too intriguing
not to share it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia