[Python-ideas] PEP 511: API for code transformers

[Andrew Barnert]

On Thursday, January 28, 2016 5:07 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> > On Thu, Jan 28, 2016 at 09:13:08PM +0000, Andrew Barnert via Python-ideas wrote:
> 
>>  This kind of talk worries me. It's _already_ very easy to write AST 
>>  transformers. There's no need for any third-party code from PyPI, and 
>>  that "your own solution" that you have to write is a few lines of 
> 
>>  trivial code.
>> 
>>  I think a lot of people don't realize this.
> 
> I don't realise this.
> 
> Not that I don't believe you, but I'd like to see a tutorial that goes 
> through this step by step and actually explains what this is all about. 
> Or, if it really is just a matter of a few lines, even just a simple 
> example might help.


I agree, but someone (Brett?) on one of these threads explained that they don't include such a tutorial in the docs because they don't want to encourage people to screw around with import hooks too much, so...

Anyway, I wrote a blog post about last year (
http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking-python.html), but I'll summarize it here. I'll show the simplest code for hooking in a source, AST, or bytecode transformer, not the most production-ready.

> For instance, the PEP includes a transformer that changes all string 

> literals to "Ni! Ni! Ni!". Obviously it doesn't work as 
> sys.set_code_transformers doesn't exist yet, but if I'm understanding 
> you, we don't need that because it's already easy to apply that 
> transformer. Can you show how? Something that works today?


Sure. Here's an AST transformer:

    class NiTransformer(ast.NodeTransformer):
        def visit_Str(self, node):
            node.s = 'Ni! Ni! Ni!'
            return node
    
Here's a complete loader implementation that uses the hook:

    class NiLoader(importlib.machinery.SourceFileLoader):
        def source_to_code(self, data, path, *, _optimize=-1):
            source = importlib._bootstrap.decode_source(data)
            tree = NiTransformer().visit(ast.parse(source, path, 'exec'))

            return compile(tree, path, 'exec')


Now, how do you install the hook? That depends on what exactly you want to do. Let's say you want to make it globally hook all .py files, be transparent to .pyc generation, and ignore -O, and you'd prefer a monkeypatch hack that works on all versions 3.3+, rather than a clean spec-based finder that requires 3.5. Here goes:

    finder = sys.meta_path[-1]
    loader = finder.find_module(__file__)
    loader.source_to_code = NiLoader.source_to_code


Just put all this code in your top level script, or just put it in a module and import that in your top level script, either way before importing anything else. (And yes, "before importing anything else" means some bits of the stdlib end up processed and some don't, just as with PEP 511.) You can see it in action at https://github.com/abarnert/nihack

PEP 511 writes the NiLoader part for you, but, as you can see, that's the easiest part of the whole thing.

If you want all the exact same choices that the PEP makes (global, .py files only, insert name into .pyc files, integrate with -O and -o, promise to be semantically neutral, etc.), it also makes the last part trivial, which is a much bigger deal. If you want any different choices, it doesn't help with the last part at all. (And I think that's fine, as long as that's the intention. Right now, someone has to have some idea of what they're doing to use my hack, and that's probably a good thing, right? And if I want to clean it up and make it distributable, like MacroPy, I'd better know how to write a spec finder or I have no business distributing any such thing. But if people want to experiment with optimizers that don't actually change the behavior of their code, that's a lot safer, so it seems reasonable that we should focus on making that easier.)