[Python-ideas] PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not

[Petr Viktorin]

On 01/27/2016 04:39 PM, Victor Stinner wrote:
> Hi,
> 
> Thank you for all feedback on my PEP 511. It looks like the current
> blocker point is the unclear status of "language extensions": code
> tranformers which deliberately changes the Python semantics. I would
> like to discuss how we should register them. I think that the PEP 511
> must discuss "language extensions" even if it doesn't have to propose
> a solution to make their usage easier. It's an obvious usage of code
> transformers. If possible, I would like to find a compromise to
> support them, but make it explicit that they change the Python
> semantics.
> 
> By the way, I discussed with Joseph Jevnik who wrote codetransformer
> (bytecode transformer) and lazy_python (AST transformer). He wrote me:
> 
> "One concern that I have though is that transformers are registered
> globally. I think that the decorators in codetransformer do a good job
> of signalling to reader the scope of some new code generation."
> 
> 
> Currently, the PEP 511 doesn't provide a way to register a code
> transformer but only use it under some conditions. For example, if
> fatoptimizer is registered, all .pyc files will be called
> file.cpython-36.fat-0.pyc even if fatoptimizer was disabled.
> 
> I propose to change the design of sys.set_code_transformers() to use
> it more like a registry similar to the codecs registry
> (codecs.register), but different (details below). A difference is that
> the codecs registry uses a mapping (codec name => codec functions),
> whereas sys.set_code_transformers() uses an ordered sequence (list) of
> code transformers. A sequence is used because multiple code
> transformers can be applied sequentially on a single .py file.
> 
> 
> Petr Viktorin wrote that language extensions "target specific modules,
> with which they're closely coupled: The modules won't run without the
> transformer. And with other modules, the transformer either does
> nothing (as with MacroPy, hopefully), or would fail altogether (as
> with Hy). So, they would benefit from specific packages opting in. The
> effects of enabling them globally range from inefficiency (MacroPy) to
> failures or needing workarounds (Hy)."
> 
> 
> Problem (A): solutions proposed below don't make code tranformers
> mandatory. If a code *requires* a code transformer and the code
> transformer is not registered, Python doesn't complain. Do you think
> that it is a real issue in practice? For MacroPy, it's not a problem
> in practice since functions must be decorated using a decorator from
> the macropy package. If importing macropy fails, the module cannot be
> imported.
> 
> 
> Problem (B): proposed solutions below adds markers to ask to enable a
> specific code transformer, but a code transformer can decide to always
> modify the Python semantics without using such marker. According to
> Nick Coghlan, code transformers changing the Python semantics *must*
> require a marker in the code using them. IMHO it's the responsability
> of the author of the code transformer to use markers, not the
> responsability of Python.

I believe Nick meant that if a transformer modifies semantics of
un-marked code, it would be considered a badly written transformer that
doesn't play well with the rest of the language.
The responsibility of Python is just to make it easy to do the right thing.

> Code transformers should maybe return a flag telling if they changed
> the code or not. I prefer a flag rather than comparing the output to
> the input, since the comparison can be expensive, especially for a
> deep AST tree. Example:
> 
> class Optimizer:
>     def ast_optimizer(self, tree, context):
>         # ...
>         return modified, tree
> 
> *modified* must be True if tree was modified.

What would this flag be useful for?

> There are several options to decide if a code transformer must be used
> on a specific source file.
> 
> 
[...]
> (2) Petr proposed to extend importlib to pass a code transformer when
> importing a module.
> 
>     importlib.util.import_with_transformer(
>         'mypackage.specialmodule', MyTransformer())
> 
> 
[...]
> (5) Nick proposed (indirectly) to use a different filename (don't use
> ".py") for language extensions.
> 
> This option works with my option (2): the context contains the
> filename which can be used to decide to enable or not the code
> transformer.
> 
> I understand that the code transformer must also install an importlib
> hook to search for other filenames than only .py files. Am I right?

Yes, you are. But once a custom import hook is in place, you can just
use a regular import, the hack in (2) isn't necessary.

Also, note that this would solve problem (A) -- without the hook
enabled, the source won't be found.