[Python-ideas] PEP 511: API for code transformers

[Andrew Barnert]

On Thursday, January 28, 2016 7:10 PM, Andrew Barnert <abarnert at yahoo.com> wrote:


<snip>
Immediately after sending that, I realized that Victor's PEP uses a bytecode transform rather than an AST transform. That isn't much harder to do today. Here's a quick, untested version:

    def ni_transform(c):
        consts = []
        for const in c.co_consts:
            if isinstance(c, str):
                consts.append('Ni! Ni! Ni!')
            elif isinstance(c, types.CodeType):
                consts.append(ni_transform(const))
            else:
                consts.append(const)
        return types.CodeType(
            c.co_argcount, c.co_kwonlyargcount, c.co_nlocals, c.co_stacksize,
            c.co_flags, c.co_code, tuple(consts), c.co_names, c.co_varnames,
            c.co_filename, c.co_name, c.co_firstlineno, c.co_lnotab,
            c.co_freevars, c.co_cellvars)

    class NiLoader(importlib.machinery.SourceFileLoader):
        def source_to_code(self, data, path, *, _optimize=-1):
            return ni_transform(compile(data, path, 'exec'))

You may still need the decode_source bit, at least on some of the Python versions; I can't remember. If so, add that one line from the AST version.


Installing the hook is the same as the AST version.

You may notice that I have that horrible 18-argument constructor, and the PEP doesn't. But that's because the PEP is basically cheating with this example. For some reason, it passes 3 of those arguments separately--consts, names, and lnotab. If you modify anything else, you'll need the same horrible constructor. And, in any realistic bytecode transformer, you will need to modify something else. For example, you may want to transform the bytecode.

And meanwhile, once you start actually transforming bytecode, that becomes the hard part, and PEP 511 won't help you there. If you just want to replace every LOAD_GLOBAL with a LOAD_CONST, you can do that in a pretty simple loop with a bit of help from the dis module. But if you want to insert and delete bytecodes like the existing peephole optimizer in C does, then you're also dealing with renumbering jump targets and rebuilding the lnotab and other fun things. And if you start dealing with opcodes that change the stack effect nonlocally, like with and finally handlers, you'd have to be an idiot or a masochist to not reach for a third-party library like byteplay. (I know this because I'm enough of an idiot to have done it once, but not enough of an idiot or a masochist to do it again...).

So, again, PEP 511 isn't helping with the hard part. But, again, I think that may be fine. (Someone who knows how to use byteplay well enough to build a semantically-neutral optimizer function decorator, I'll trust him to be able to turn that into a global optimizer with one line of code. But if he wants to hook things in transparently to .pyc files, or to provide actual language extensions, or something like that, I think it's OK to make him do a bit more work before he can give it to me as production-ready code.)