[Python-ideas] Enabling access to the AST for Python code

[Brett Cannon]

On Thu, May 21, 2015 at 10:10 PM Ben Hoyt <benhoyt at gmail.com> wrote:

> Huh, interesting idea. I've never used import hooks. Looks like the
> relevant macropy source code is here:
>
> https://github.com/lihaoyi/macropy/blob/master/macropy/core/import_hooks.py
>
> So basically you would do the following:
>
> 1) intercept the import
> 2) find the source code file yourself and read it
> 3) call ast.parse() on the source string
> 4) do anything you want to the AST, for example turn the "select(c for
> c in Customer if sum(c.orders.price) > 1000" into whatever SQL or
> other function calls
> 5) pass the massaged AST to compile(), execute it and return the module
>
> Hmmm, yeah, I think you're basically suggesting macro-like processing
> of the AST. Pretty cool, but not quite what I was thinking of ... I
> was thinking select() would get an AST object at runtime and do stuff
> with it.
>

Depending on what version of Python you are targeting, it's actually
simpler than that even to get it into the import system:

   1. Subclass importlib.machinery.SourceFileLoader
   <https://docs.python.org/3/library/importlib.html#importlib.machinery.SourceFileLoader>
   and override source_to_code()
   <https://docs.python.org/3/library/importlib.html#importlib.abc.InspectLoader.source_to_code>
   to do your AST transformation and return your changed code object
   (basically your steps 3-5 above)
   2. Set a path hook that uses an instance of
   importlib.machinery.FileFinder
   <https://docs.python.org/3/library/importlib.html#importlib.machinery.FileFinder>
   which utilizes your custom loader
   3. There is no step 3

I know this isn't what you're after, but I just wanted to let you know
importlib has made this sort of thing fairly trivial to implement.

-Brett


>
> -Ben
>
> On Thu, May 21, 2015 at 9:51 PM, Andrew Barnert <abarnert at yahoo.com>
> wrote:
> > On May 21, 2015, at 18:18, Ben Hoyt <benhoyt at gmail.com> wrote:
> >>
> >> (I know that there's the "ast" module and ast.parse(), which can give
> >> you an AST given a *source string*, but that's not very convenient
> >> here.)
> >
> > Why not? Python modules are distributed as source. You can pretty easily
> write an import hook to intercept module loading at the AST level and
> transform it however you want. Or just use MacroPy, which wraps up all the
> hard stuff (especially 2.x compatibility) and provides a huge framework of
> useful tools. What do you want to do that can't be done that way?
> >
> > For many uses, you don't even have to go that far--code objects remember
> their source file and line number, which you can usually use to retrieve
> the text and regenerate the AST.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150522/dd242a44/attachment.html>