[Python-ideas] PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not

[Sven R. Kunze]

Hi,

On 27.01.2016 16:39, Victor Stinner wrote:
> "One concern that I have though is that transformers are registered
> globally. I think that the decorators in codetransformer do a good job
> of signalling to reader the scope of some new code generation."

I share this concern but haven't a good solution right now. Admittedly, 
I already have a use-case where I would like to apply a transformation 
which is NOT an optimization but a global extension.

So, the discussion about allowing global extension really made me think 
about whether that is really a good idea. *BUT* it would allow me to 
experiment and find out if the risk is worth it.

(use-case: adding some hooks before entering and leaving all try blocks)

> Currently, the PEP 511 doesn't provide a way to register a code
> transformer but only use it under some conditions. For example, if
> fatoptimizer is registered, all .pyc files will be called
> file.cpython-36.fat-0.pyc even if fatoptimizer was disabled.
>
> I propose to change the design of sys.set_code_transformers() to use
> it more like a registry similar to the codecs registry
> (codecs.register), but different (details below). A difference is that
> the codecs registry uses a mapping (codec name => codec functions),
> whereas sys.set_code_transformers() uses an ordered sequence (list) of
> code transformers. A sequence is used because multiple code
> transformers can be applied sequentially on a single .py file.

How does it change the interface for the users? (I mean besides the 
renaming).



I still like your idea of having the following three options:

1) global optimizers
2) local extensions --> via codec or import hook
3) global extension --> use with care


So, I assume we talk about specifying 2).

> Petr Viktorin wrote that language extensions "target specific modules,
> with which they're closely coupled: The modules won't run without the
> transformer. And with other modules, the transformer either does
> nothing (as with MacroPy, hopefully), or would fail altogether (as
> with Hy). So, they would benefit from specific packages opting in. The
> effects of enabling them globally range from inefficiency (MacroPy) to
> failures or needing workarounds (Hy)."
>
>
> Problem (A): solutions proposed below don't make code tranformers
> mandatory. If a code *requires* a code transformer and the code
> transformer is not registered, Python doesn't complain. Do you think
> that it is a real issue in practice? For MacroPy, it's not a problem
> in practice since functions must be decorated using a decorator from
> the macropy package. If importing macropy fails, the module cannot be
> imported.

Sounds good.

> Problem (B): proposed solutions below adds markers to ask to enable a
> specific code transformer, but a code transformer can decide to always
> modify the Python semantics without using such marker. According to
> Nick Coghlan, code transformers changing the Python semantics *must*
> require a marker in the code using them. IMHO it's the responsability
> of the author of the code transformer to use markers, not the
> responsability of Python.

I agree with Nick. Be explicit.

> Code transformers should maybe return a flag telling if they changed
> the code or not. I prefer a flag rather than comparing the output to
> the input, since the comparison can be expensive, especially for a
> deep AST tree. Example:
>
> class Optimizer:
>      def ast_optimizer(self, tree, context):
>          # ...
>          return modified, tree
>
> *modified* must be True if tree was modified.

Not sure if that is needed. If we don't have an immediate use-case, 
simpler is better.

> There are several options to decide if a code transformer must be used
> on a specific source file.

The user should decide, otherwise there is too much magic involved: a 
marker (source file) or an option (cmdline).

I am indifferent whether the marker should be a codec-decl or an import 
hook. But it should be file-local (at least I would prefer that).


All of the options below seem to involve too much magic for my taste (or 
I didn't understand them correctly).

> (1) Add a check_code() and check_ast() functions to code transformers.
> The code transformer is responsible to decide if it wants to transform
> the code or not. Python doesn't use the code transformer if the check
> method returns False.
>
> Examples:
>
> * MacroPy can search for the "import macropy" statement (of "from
> macropy import ...") in the AST tree
> * fatoptimizer can search for "__fatoptimizer__ = {'enabled': False}"
> in the code: if this variable is found, the optimizer is completly
> skipped
>
>
> (2) Petr proposed to extend importlib to pass a code transformer when
> importing a module.
>
>      importlib.util.import_with_transformer(
>          'mypackage.specialmodule', MyTransformer())
>
> IMHO this option is too specific: it's restricted to importlib
> (py_compile, compileall and interactive interpreter don't have the
> feature). I also dislike the API.
>
>
> (3) Petr also proposed "a special flag in packages":
>
>      __transformers_for_submodules__ = [MyTransformer()]
>
> I don't like having to get access to MyTransformer. The PEP 511
> mentions an use case where the transformed code is run *without*
> registering the transformer. But this issue can easily be fixed by
> using the string to identify the transformer in the registery (ex:
> "fat") rather than its class.
>
> I'm not sure that putting a flag on the package (package/__init__.py?)
> is a good idea. I would prefer to enable language extensions on
> individual files to restrict their scope.
>
>
> (4) Sjoerd Job Postmus proposed something similar but using a comment
> and not for packages, but any source file:
>
>      #:Transformers modname.TransformerClassName,
> modname.OtherTransformerClassName
>
> The problem is that comments are not stored in the AST tree. I would
> prefer to use AST to decide if an AST transformer should be used or
> not.
>
> Note: I'm not really motived to extend the AST to start to include
> comments, or even code formatting (spaces, newlines, etc.).
> https://pypi.python.org/pypi/redbaron/ can be used if you want to
> transform a .py file without touching the format. But I don't think
> that AST must go to this direction. I prefer to keep AST simple.
>
>
> (5) Nick proposed (indirectly) to use a different filename (don't use
> ".py") for language extensions.
>
> This option works with my option (2): the context contains the
> filename which can be used to decide to enable or not the code
> transformer.
>
> I understand that the code transformer must also install an importlib
> hook to search for other filenames than only .py files. Am I right?
>
>
> (6) Nick proposed (indirectly) to use an encoding cookie "which are
> visible as a comment in the module header".
>
> Again, I dislike this option because comments are not stored in AST.

Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160127/3b651862/attachment-0001.html>