[Python-ideas] PEP 511: API for code transformers

[Andrew Barnert]

On Jan 16, 2016, at 19:49, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Making that distinction means we can be clear that the transformation
> case is already well served by import hooks that process alternate
> filename extensions rather than standard Python source or bytecode
> files, encoding cookie tricks (which are visible as a comment in the
> module header), and function decorators that alter the semantics of
> the functions they're applied to.
> 
> The case which *isn't* currently well served is transparently applying
> a semantics preserving code optimiser like FAT Python - that's a
> decision for the person *running* the code, rather than the person
> writing it...

I think something that isn't made clear in the rationale is why an import hook is good enough for most semantic extensions, but isn't good enough for global optimizers.

After all, it's not that hard to write a module that installs an import hook for normal .py files instead of .hy or .pyq or whatever files. Then, to optimize your own code, or a third-party library, you just import the optimizer module first; to optimize an application, you write a 2- or 3-line wrapper (which can be trivially automated a la setuptools entry point scripts) to import the optimizer and then start the app.

There are good reasons that isn't sufficient. For example, parts of the stdlib have already been imported before the top of the main module. While there are ways around that (I believe FAT comes with a script to recompile the stdlib into a venv or something?), they're clumsy and ad hoc, and it's unlikely two different optimizers would play nicely together. Also, making it work in a sensible way with .pyc files takes a decent amount of code, and will again be an ad-hoc solution that won't play well with other projects doing similar things. And there are people who write and execute long-running, optimization-ripe bits of code in the REPL (or at least in an IPython notebook), and that can't be handled with an import hook. Nor can code that extensively used exec. And probably other reasons I haven't thought of.

Maybe the PEP should explain those reasons, so it's clear why this feature will help projects like FAT.

Then again, some of those same reasons seem to apply equally well to semantic extensions. Two extensions are no more likely to play together as import hooks than two optimizers, and yet in many cases there's no syntactic or semantic reason they couldn't. Extensions are probably even more useful than optimizations at the REPL. And so on. And this is all even more true for extensions that people write to explore a new feature idea than for things people want to publish as deployable code.

So, I'm still not convinced that the distinction really is critical here. I definitely don't see why it's a negative that the PEP can serve both purposes, even if people only want one of them served (normally, Python doesn't go out of its way to prevent writing certain kinds of code, it just becomes accepted that such code is not idiomatic; only when there's a real danger of attractive nuisance is the language modified to ban it), and I think it's potentially a positive.