[Python-ideas] PEP 511: API for code transformers

[Kevin Conway]

I'm a big fan of your motivation to build an optimizer for cPython code.
What I'm struggling with is understanding why this requires a PEP and
language modification. There are already several projects that manipulate
the AST for performance gains such as [1] or even my own ham fisted attempt
[2].

Would you please elaborate on why these external approaches fail and how
language modifications would make your approach successful.

[1] https://pypi.python.org/pypi/astoptimizer
[2] http://pycc.readthedocs.org/en/latest/

On Sat, Jan 16, 2016, 10:30 Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:

> On Sat, Jan 16, 2016 at 12:06:58PM +0100, Petr Viktorin wrote:
> > The "transformer" API can be used for syntax extensions as well, but the
> > registration needs to be different so the effects are localized. For
> > example it could be something like::
> >
> >     importlib.util.import_with_transformer(
> >         'mypackage.specialmodule', MyTransformer())
> >
> > or a special flag in packages::
> >
> >     __transformers_for_submodules__ = [MyTransformer()]
> >
> > or extendeding exec (which you actually might want to add to the PEP, to
> > make giving examples easier)::
> >
> >     exec("print('Hello World!')", transformers=[MyTransformer()])
> >
> > or making it easier to write an import hook with them, etc...
>
> So, you'd have to supply the transformer used before importing? That
> seems like a troublesome solution to me.
>
> A better approach (to me) would require being able to document what
> transformers need to be run inside the module itself. Something like
>
>     #:Transformers modname.TransformerClassName,
> modname.OtherTransformerClassName
>
> The reason why I would prefer this, is that it makes sense to document
> the transformers needed in the module itself, instead of in the code
> importing the module.
>
> As you suggest (and rightly so) to localize the effects of the
> registration, it makes sense to do the registration in the affected
> module.
>
> Of course there might be some cases where you want to import a module
> using a transformer it does not need to know about, but I think that
> would be less likely than the case where a module knows what
> transformers there should be applied.
>
> As an added bonus, it would let you apply transformers to the
> entry-point:
>
>     #!/usr/bin/env python
>     #:Transformers foo.BarTransformerMyCodeCanNotRunWithout
>
> But as you said, this support is probably outside the scope of the PEP
> anyway.
>
> Kind regards,
> Sjoerd Job
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160116/d50e107a/attachment.html>