[Python-ideas] PEP 511: API for code transformers

[Andrew Barnert]

On Jan 17, 2016, at 03:48, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> 2016-01-16 12:06 GMT+01:00 Petr Viktorin <encukou at gmail.com>:
> 
> > The PEP is designed optimizers. It would be good to stick to that use
> > case, at least as far as the registration is concerned. I suggest noting
> > in the documentation that Python semantics *must* be preserved, and
> > renaming the API, e.g.::
> >
> >     sys.set_global_optimizers([])
> 
> I would prefer to not restrict the PEP to a specific usage.
> 
> > The "transformer" API can be used for syntax extensions as well, but the
> > registration needs to be different so the effects are localized. For
> > example it could be something like::
> >
> >     importlib.util.import_with_transformer(
> >         'mypackage.specialmodule', MyTransformer())
> 
> Brett may help on this part. I don't think that it's the best way to use importlib. importlib is already pluggable. As I wrote in the PEP, MacroPy uses an import hook. (Maybe it should continue to use an import hook?)
> 
> > or a special flag in packages::
> >
> >     __transformers_for_submodules__ = [MyTransformer()]
> 
> Does it mean that you have to parse a .py file to then decide how to transform it? It will slow down compilation of code not using transformers.
> 
> I would prefer to do that differently: always register transformers very early, but configure each transformer to only apply it on some files.

At that point, you're exactly duplicating what can be done with import hooks.

I think this is part of the reason Nick suggested the PEP should largely ignore the issue of syntax extensions and experiments: because then you don't have to solve Petr's problem. Globally-applicable optimizers are either on or off globally, so the only API you need to control them is a simple global list. The fact that this same API works for some uses of extensions doesn't matter; the fact that it doesn't work for some other uses of extensions also doesn't matter; just design it for the intended use.

> The transformer can use the filename (file extension? importlib is currently restricted to .py files by default no?),

Everything goes through the same import machinery. The usual importer gets registered for .py files. Something like hylang can register for a different extension. Something like PyMacro can wrap the usual importer, then register to take over for .py files. (This isn't quite what PyMacro does, because it's designed to work with older versions of Python, with less powerful/simple customization opportunities, but it's what a new PyMacro-like project would do.) A global optimizer could also be written that way today. And doing this is a couple dozen lines of code (or about 5 lines to do it as a quick&dirty hack without worrying about portability or backward/forward compatibility).

The reason your PEP is necessary, I believe, is to overcome the limitations of such an import hook: to work at the REPL/notebook/etc. level, to allow multiple optimizers to play nicely without them having to agree on some wrapping protocol, to work with exec, etc. By keeping things simple and only serving the global case, you can (or, rather, you already have) come up with easier solutions to those issues--no need for enabling/disabling files by type or other information, no need for extra optional parameters to exec, etc.

(Or, if you aren't trying to overcome those limitations, then I'm not sure why your PEP is necessary. Import hooks already work, after all.)

> > Another thing: this snippet from the PEP sounds too verbose::
> >
> >     transformers = sys.get_code_transformers()
> >     transformers.insert(0, new_cool_transformer)
> >     sys.set_code_transformers(transformers)
> >
> > Can this just be a list, as with sys.path? Using the "optimizers" term::
> >
> >     sys.global_optimizers.insert(0, new_cool_transformer)
> 
> set_code_transformers() checks the transformer name and ensures that the transformer has at least a AST transformer or a bytecode transformer. That's why it's a function and not a simple list.

That doesn't seem necessary. After all, sys.path doesn't check that you aren't assigning non-strings or strings that don't make valid paths, and nobody has ever complained that it's too hard to debug the case where you write `sys.paths.insert(0, {1, 2, 3})` because the error comes at import time instead of locally.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160117/5cbe722a/attachment.html>