[Python-checkins] peps: PEP 511: add the text of the PEP

[Terry Reedy]

On 1/12/2016 7:26 PM, victor.stinner wrote:

>    PEP 511: add the text of the PEP

>   Abstract
>   ========
>
> -Propose an API to support AST transformers.
> +Propose an API to support AST transformers. Add also ``-o OPTIM_TAG``
> +command line option to change ``.pyc`` filenames. Raise an
> +``ImportError`` exception on import if the ``.pyc`` file is missing and
> +the AST transformers required to transform the code are missing.

> +AST transformers are not needed code transformed ahead of time (loaded
> +from ``.pyc`` files).

This sentence is garbled.

> +Transforming the code allows to extend the Python language for specific
> +use cases. Transforming an Abstract Syntax Tree (AST) is a convenient
> +way to implement an optimizer. It's easier to work on the AST than
> +working on the bytecode, AST contains more information and is more high
> +level.

For many things, it should also be easier to work on the AST than the 
source code.  But, it would be nice to have an AST tree pattern matcher, 
with capture groups, equivalent to re for linear text.

> +This PEP proposes to add an API to register AST transformers.
> +
> +A new ``-o OPTIM_TAG`` command line option is added to only load
> +transformed code: it changes the name of searched ``.pyc`` files. If the
> +``.pyc`` file of a module is missing and the ``.py`` is available, an
> +``ImportError`` exception is raised import if the AST transformers

'raised on import'

> +required to transform the code are missing. The import behaviour with
> +the default optimizer tag (``'opt'``) is unchanged.
> +
> +The transformation can done ahead of time.

I presume you mean during the one-time compile and generate .pyc step.

 > It allows to implement powerful but expensive transformations.

For many transformations, the expensive part is finding subtrees 
matching the applicable pattern.

The Windows installer optionally runs compileall.  Another minute for an 
install would not be too bad.  In any case, stdlib modules are stable 
for months at a time.  Ditto for installations of most 3rd party 
packages. Getting months of use from one time costs, added on to 
installs that may proceed while one does something else, is an important 
point.

> +Use Cases
> +=========
> +
> +This section give examples of use cases explaining when and how AST
> +transformers will be used.
> +
> +Interactive interpreter
> +-----------------------
> +
> +It will be possible to use AST transformers with the interactive
> +interpreter which is popular in Python and commonly used to demonstrate
> +Python.

Possible, but limited to default inexpensive transformations with 
immediate payoff, such as the current peephole optimizer.

> +The code is transformed at runtime and so the interpreter can be slower
> +when expensive AST transformers are used.

Expensive transformers should be turned off.  I am not sure what you are 
saying here.

Now for some possible dumb quesitons.

> +Build a transformed package
> +---------------------------

I am probably missing some the import differences between this and

+Install a package containing transformed .pyc files
+---------------------------------------------------

and

+Build .pyc files when installing a package
+------------------------------------------

> +It will be possible to build a package of the transformed code.
> +
> +A transformer can have a configuration. The configuration is not stored
> +in the package.
> +
> +All ``.pyc`` files of the package must be transformed with the same AST
> +transformers and the same transformers configuration.

Why? is unclear to me.  The issues I have:

Suppose I write (as I might) one or more recursion-to-iteration 
transformers (within one transformer module).  Each function 
transformation would be independent of all others, so could potentially 
be applied on a function-to-function basis.  (For instance, decorators 
might be used to indicate which transformer, if any, to apply to a 
particular function.) For simple linear recursive functions (such as 
factorial), eliminating the repeated function calls might double the speed.

The grouping of modules into packages is somewhat arbitrary.  An 
application often uses modules from multiple packages, so it must anyway 
execute with different modules having different optimizations.

A transform may not be 100% reliable, or may not be believed to be so by 
the programmer, or may depend on information supplies by the programmer. 
  So a careful programmer only wants it applied after testing.  The 
boundaries of what has been tested may not coincide with package boundaries.

A related issue is that AST transformations may affect tracebacks, and a 
programmer might want to temporarily disable a transformation for one 
function, without having every module in a large package re-compiled to 
yet another set of .pyc files.  I am partly worried here about 
combinatorial explosion of space usage.

Suppose a transformer looks for certain decorators.  Running that 
transformer over files that do not contain the trigger decorators is 
harmless, in that nothing will be changed, but is also irrelevant and a 
waste of time.

> +API to support AST transformers:

> +* Add ``sys.ast_transformers``: list of AST transformers used to rewrite
> +  an AST tree. The list is empty by default: no AST transformer.

I do not understand how this would work, or rather it seems to conflict 
with 'all modules in each package must have a matching set of optimized 
.pycs'.

> +* Add ``sys.implementation.ast_transformers``: name of AST
> +  transformers registered in ``sys.ast_transformers``

How is this different from sys.ast_transformers?

...

> +AST transformer
> +----------------
> +
> +Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::

This helps a lot.

> +    import ast
> +    import sys
> +
> +
> +    class KnightsWhoSayNi(ast.NodeTransformer):
> +        def visit_Str(self, node):
> +            node.s = 'Ni! Ni! Ni!'
> +            return node

This I can do today.  Tree pattern recognition has to be hard-coded and 
substring capture done manually.

> +    class ASTTransformer:
> +        name = "knights_who_say_ni"
> +
> +        def __call__(self, tree, context):
> +            KnightsWhoSayNi().visit(tree)
> +            return tree
> +
> +
> +    # register the AST transformer
> +    sys.ast_transformers.append(ASTTransformer())

This is new, with the result

> +    # execute code which will be transformed by ast_transformer()
> +    exec("print('Hello World!')")
> +
> +Output::
> +
> +    Ni! Ni! Ni!

that this happens automatically, without having to write out
    exec(compile(Transformer.visit(compile-to-ast(code))))
More important, the transformation happens automatically for import 
statements without having to split up the import process to dig out and 
similarly augment the compile process.  That will

tjr