[Python-ideas] PEP 511: API for code transformers

[Andrew Barnert]

On Thursday, January 28, 2016 12:44 PM, Brett Cannon <brett at python.org> wrote:
>On Wed, 27 Jan 2016 at 20:57 Kevin Conway <kevinjacobconway at gmail.com> wrote:


>>Out of curiosity, do you consider this PEP as adding something to Python that didn't previously exist or do you consider this PEP to more aligned with PEP0249 (DB2API) and PEP0484 (Type Hints) which are primarily designed to marshal the community in a common direction? I understand that you have other PEPs in flight that are designed to make certain optimizations easier (or possible). Looking at this PEP in isolation, however, leaves me wanting more explanation as to its value.>

>The PEP is about empowering people to write AST transformers without having to use third-party tools to integrate it into their workflow. As you pointed out, there is very little here that isn't possible today with some toolchain that reads Python source code, translates it into an AST, optimizes it, and then writes out the .pyc file. But that all does require going to PyPI or writing your own solution.

This kind of talk worries me. It's _already_ very easy to write AST transformers. There's no need for any third-party code from PyPI, and that "your own solution" that you have to write is a few lines of trivial code.

I think a lot of people don't realize this. Maybe because they tried it in 2.6 or 3.2, where it was a lot harder, or because they read the source to MacroPy (which is compatible with 2.6 and 3.2, or at least originally was), where it looks very hard, or maybe just because they didn't realize how much work has already been put in to make it easy. But whatever the reason, they're wrong. And so they're expecting this PEP to solve a problem that doesn't need to be solved.

>But if Victor's PEP gets in, then there will be a standard hook point that all Python code will go through which will make adding AST transformers much easier. Whether this ease of use is beneficial is part of the discussion around this PEP.

There already is a standard hook point that all Python code goes through. Writing an AST transformer is as simple as replacing the code that compiles source to bytecode with a 3-line function that compiles source to AST, calls your transformer, and compiles AST to bytecode. Processing source or bytecode instead of AST is just as easy (actually, one line shorter).

Where it gets tricky is all the different variations on what you hook and how. Do you want to intercept all .py files? Or add a new extension, like .hy, instead? Or all source files, but only if they start with a magic marker line? How do you want to integrate with naming, finding, obsoleting, reading, and writing .pyc files? What about -O? And so on. And how do you want to work together with other libraries trying to do the same thing, which may have made slightly different decisions? Once you decide what you want, it's another few lines to write and install the hook that does that--the hard part is deciding what you want.

If this PEP can solve the hard part in a general way, so that the right thing to do for different kinds of transformers will be obvious and easy, that would be great.

If it can't do so, then it just shouldn't bother with anything that doesn't fit into its model of global semantic-free transformations. And that would also be great--making global semantic-free transformers easy is already a huge boon even if it doesn't do anything else, and keeping the design for that as simple as possible is better than making it more complex to partially solve other things in a way that only helps with the easiest parts.