[Python-checkins] peps: PEP 511
victor.stinner
python-checkins at python.org
Thu Jan 14 19:45:20 EST 2016
https://hg.python.org/peps/rev/10df1af7e4a1
changeset: 6185:10df1af7e4a1
user: Victor Stinner <victor.stinner at gmail.com>
date: Fri Jan 15 01:44:58 2016 +0100
summary:
PEP 511
* Fix the API: use sys.get/set_ast_transformers()
* elaborate changes on importlib, try to be more concrete
* add more examples of usage of AST transformers
files:
pep-0511.txt | 150 +++++++++++++++++++++++++++-----------
1 files changed, 107 insertions(+), 43 deletions(-)
diff --git a/pep-0511.txt b/pep-0511.txt
--- a/pep-0511.txt
+++ b/pep-0511.txt
@@ -37,15 +37,40 @@
Usage 1: AST optimizer
----------------------
+Python 3.6 optimizes the code using a peephole optimizer. By
+definition, a peephole optimizer has a narrow view of the code and so
+can only implement basic optimizations. The optimizer rewrites the
+bytecode. It is difficult to enhance it, because it written in C.
+
Transforming an Abstract Syntax Tree (AST) is a convenient
way to implement an optimizer. It's easier to work on the AST than
working on the bytecode, AST contains more information and is more high
level.
-Python 3.6 optimizes the code using a peephole optimizer. By
-definition, a peephole optimizer has a narrow view of the code and so
-can only implement basic optimizations. The optimizer rewrites the
-bytecode. It is difficult to enhance it, because it written in C.
+Example of optimizations which can be implemented with an AST optimizer:
+
+* `Copy propagation
+ <https://en.wikipedia.org/wiki/Copy_propagation>`_:
+ replace ``x=1; y=x`` with ``x=1; y=1``
+* `Constant folding
+ <https://en.wikipedia.org/wiki/Constant_folding>`_:
+ replace ``1+1`` with ``2``
+* `Dead code elimination
+ <https://en.wikipedia.org/wiki/Dead_code_elimination>`_
+
+Using guards (see the PEP 510), it is possible to implement a much wider choice
+of optimizations. Examples:
+
+* Simplify iterable: replace ``range(3)`` with ``(0, 1, 2)`` when used
+ as iterable
+* `Loop unrolling <https://en.wikipedia.org/wiki/Loop_unrolling>`_
+* Call pure builtins: replace ``len("abc")`` with ``3``
+* Copy used builtin symbols to constants
+
+See also `optimizations implemented in fatoptimizer
+<https://fatoptimizer.readthedocs.org/en/latest/optimizations.html>`_, a
+static optimizer for Python 3.6.
+
Usage 2: Preprocessor
---------------------
@@ -53,14 +78,17 @@
A preprocessor can be easily implemented with an AST transformer. A
preprocessor has various and different usages. Examples:
-* Remove debug code (like assertions and logs) to make the code faster to run
+* Remove debug code like assertions and logs to make the code faster to run
it for production.
* `Tail-call Optimization <https://en.wikipedia.org/wiki/Tail_call>`_
* Add profiling code
-* Lazy macro create a memoizing thunk.
-
-Examples extending or changing the Python language:
-
+* `Lazy evaluation <https://en.wikipedia.org/wiki/Lazy_evaluation>`_:
+ see `lazy_python <https://github.com/llllllllll/lazy_python>`_
+ (bytecode transformer) and `lazy macro of MacroPy
+ <https://github.com/lihaoyi/macropy#lazy>`_ (AST transformer)
+* Change dictionary literals into collection.OrderedDict instances
+* Declare constants: see `@asconstants of codetransformer
+ <https://pypi.python.org/pypi/codetransformer>`_
* Domain Specific Language (DSL) like SQL queries. The
Python language itself doesn't need to be modified. Previous attempts to
implement DSL for SQL like `PEP 335 - Overloadable Boolean Operators
@@ -69,7 +97,8 @@
* String Interpolation, but `PEP 498 -- Literal String Interpolation
<https://www.python.org/dev/peps/pep-0498/>`_ was merged into Python 3.6.
-MacroPy has a much longer list of examples and use cases.
+`MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of
+examples and use cases.
Use Cases
@@ -147,38 +176,25 @@
This PEP proposes to add an API to register AST transformers.
-A new ``-o OPTIM_TAG`` command line option is added to only load
-transformed code: it changes the name of searched ``.pyc`` files. If the
-``.pyc`` file of a module is missing and the ``.py`` is available, an
-``ImportError`` exception is raised import if the AST transformers
-required to transform the code are missing. The import behaviour with
-the default optimizer tag (``'opt'``) is unchanged.
-
The transformation can done ahead of time. It allows to implement
powerful but expensive transformations.
-API to support AST transformers:
-* Add ``sys.ast_transformers``: list of AST transformers used to rewrite
- an AST tree. The list is empty by default: no AST transformer.
-* Add ``sys.implementation.ast_transformers``: name of AST
- transformers registered in ``sys.ast_transformers``
-* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
- The default optimization tag is ``'opt'``.
-* Use the optimizer tag in ``.pyc`` filenames in ``importlib``.
- Remove also the special case for the optimizer level ``0`` with the
- default optimizer tag ``'opt'`` to simplify the code.
-* Add a new ``-o OPTIM_TAG`` command line option to set
- ``sys.implementation.optim_tag``
+API for AST transformers
+------------------------
-.. note::
- FIXME: There is a design issue: ``sys.ast_transformers`` and
- ``sys.implementation.ast_transformers`` can be inconsistent.
- ``sys.implementation.ast_transformers`` is required at runtime in
- some corner cases to have specific code depending if a specific AST
- transformer was used. Do you have a better suggestion?
+Add new functions to register AST transformers:
-API of an AST transformer (from ``sys.ast_transformers``):
+* ``sys.set_ast_transformers(transformers)``: set the list of AST
+ transformers
+* ``sys.get_ast_transformers()``: get the list of AST
+ transformers.
+
+The order of AST transformers matter. Running transformer A and then
+transformer B can give a different output than running transformer B an
+then transformer A.
+
+API of an AST transformer:
* An AST transformer is a callable object with the prototype::
@@ -202,9 +218,55 @@
.. note::
It would be nice to pass the fully qualified name of a module in the
*context* when an AST transformer is used to transform a module, but
- it looks like the information is not available currently.
+ it looks like the information is not available in
+ ``PyParser_ASTFromStringObject()``.
-AST transformer changes:
+
+Optimizer tag
+-------------
+
+Changes:
+
+* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
+ The default optimization tag is ``'opt'``.
+* Add a new ``-o OPTIM_TAG`` command line option to set
+ ``sys.implementation.optim_tag``
+
+Changes on ``importlib``:
+
+* ``importlib`` uses ``sys.implementation.optim_tag`` to build the
+ ``.pyc`` filename to importing modules, instead of always using
+ ``opt``. Remove also the special case for the optimizer level ``0``
+ with the default optimizer tag ``'opt'`` to simplify the code.
+* When loading a module, if the ``.pyc`` file is missing but the ``.py``
+ is available, the ``.py`` is only used if AST optimizers have the same
+ optimizer tag than the current tag, otherwise an ``ImportError``
+ exception is raised.
+
+Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can
+be compiled to import a module::
+
+ def get_ast_optim_tag():
+ transformers = sys.get_ast_transformers()
+ if not transformers:
+ return 'opt'
+ return '-'.join(transformer.name for transformer in transformers)
+
+ def use_py():
+ return (get_ast_transformers() == sys.implementation.optim_tag)
+
+The order of ``sys.get_ast_transformers()`` matter. For example, the
+``fat`` transformer followed by the ``pythran`` transformer gives the
+optimizer tag ``fat-pythran``.
+
+The behaviour of the ``importlib`` module is unchanged with the default
+optimizer tag (``'opt'``).
+
+
+AST enhancements
+----------------
+
+Enhancements to simplify the implementation of AST transformers:
* Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the
transformed AST. ``PyCF_ONLY_AST`` returns the AST before the
@@ -215,9 +277,8 @@
frozenset items.
* ``PyCodeObject.co_lnotab``: line number delta becomes signed to
support moving instructions (note: need to modify MAGIC_NUMBER in
- importlib). Implemented in the `issue #26107: code.co_lnotab: use
- signed line number delta to support moving instructions in an
- optimizer <https://bugs.python.org/issue26107>`_
+ importlib). Implemented in the `issue #26107
+ <https://bugs.python.org/issue26107>`_
* Enhance the bytecode compiler to support ``tuple`` and ``frozenset``
constants. Currently, ``tuple`` and ``frozenset`` constants are
created by the peephole transformer, after the bytecode compilation.
@@ -273,13 +334,16 @@
class ASTTransformer:
name = "knights_who_say_ni"
+ def __init__(self):
+ self.transformer = KnightsWhoSayNi()
+
def __call__(self, tree, context):
- KnightsWhoSayNi().visit(tree)
+ self.transformer.visit(tree)
return tree
# register the AST transformer
- sys.ast_transformers.append(ASTTransformer())
+ sys.set_ast_transformers([ASTTransformer()])
# execute code which will be transformed by ast_transformer()
exec("print('Hello World!')")
--
Repository URL: https://hg.python.org/peps
More information about the Python-checkins
mailing list