[Python-ideas] PEP 511: API for code transformers

[Petr Viktorin]

On 01/17/2016 12:48 PM, Victor Stinner wrote:
> 2016-01-16 12:06 GMT+01:00 Petr Viktorin <encukou at gmail.com <javascript:;>>:
>> This PEP addresses two things that would benefit from different
>> approaches: let's call them optimizers and extensions.
>>
>> Optimizers, such as your FAT, don't change Python semantics. They're
>> designed to run on *all* code, including the standard library. It makes
>> sense to register them as early in interpreter startup as possible, but
>> if they're not registered, nothing breaks (things will just be slower).
>> Experiments with future syntax (like when async/await was being
>> developed) have the same needs.
>>
>> Syntax extensions, such as MacroPy or Hy, tend to target specific
>> modules, with which they're closely coupled: The modules won't run
>> without the transformer. And with other modules, the transformer either
>> does nothing (as with MacroPy, hopefully), or would fail altogether (as
>> with Hy). So, they would benefit from specific packages opting in. The
>> effects of enabling them globally range from inefficiency (MacroPy) to
>> failures or needing workarounds (Hy).
> 
> To be clear, Hylang will not benefit from my PEP. That's why it is not
> mentioned in the PEP.
> 
> "Syntax extensions" only look like a special case of optimizers. I'm not
> sure that it's worth to make them really different.

There is an important difference: optimizers should be installed
globally. But modules that don't opt in to a specific syntax extension
should not get compiled with it.

>> The PEP is designed optimizers. It would be good to stick to that use
>> case, at least as far as the registration is concerned. I suggest noting
>> in the documentation that Python semantics *must* be preserved, and
>> renaming the API, e.g.::

My API examples seem to have led the conversation astray.
The point I wanted to make is that "syntax extensions" need a
registration API that only enables them for specific modules.

I admit the particular examples weren't very well thought out. I'm not
proposing adding *any* of them to the PEP: I'd be happy if the PEP stuck
to the "optimizers" use case and do that well.
The "extensions" case is worth another PEP, which can reuse the
transformers API (probably integrating it with importlib), but not the
registration API.

> I would prefer to do that differently: always register transformers
> very early, but configure each transformer to only apply it on some
> files. The transformer can use the filename (file extension?
> importlib is currently restricted to .py files by default no?), it
> can use a special variable in the file (ex: fatoptimizer searchs
> for a __fatoptimizer__ variable which is used to configure the
> optimizer), a configuration loaded when the transformer is
> created, etc.

Why very early? If a syntax extension is used in some package, it should
only be activated right before that package is imported. And ideally it
shouldn't get a chance to be activated on other packages.

importlib is not restricted to .py (it can do .zip, .pyc, .so, etc. out
of the box). Actually, with import hooks, the *source* file that uses
the DSL can use a different extension (as opposed to the *.pyc getting a
different tag, as for optimizers).
For example, a library using a SQL DSL could look like::

    __init__.py    (imports a package to set up the transformer)
    queries.sqlpy
    __pycache__/
        __init__.cpython-36.opt-0.pyc
        queries.cpython-36.opt-0.pyc

That is probably what you want for syntax extensions. You can't really
look at special variables in the file, because the transformer needs to
be enabled before the code is compiled -- especially if text/tokenstream
transforms are added, so the file might not be valid "vanilla Python".

What's left is making it easy to register an import hook with a specific
PEP 511 transformer -- but again, that can be a different PEP.