[Python-checkins] peps: PEP 511
victor.stinner
python-checkins at python.org
Fri Jan 15 10:52:44 EST 2016
https://hg.python.org/peps/rev/f49d35c25c49
changeset: 6190:f49d35c25c49
user: Victor Stinner <victor.stinner at gmail.com>
date: Fri Jan 15 16:18:00 2016 +0100
summary:
PEP 511
* bytecode => code
* more fun with bytecode
* enhance the Changes section
files:
pep-0511.txt | 258 ++++++++++++++++++++++++++++----------
1 files changed, 190 insertions(+), 68 deletions(-)
diff --git a/pep-0511.txt b/pep-0511.txt
--- a/pep-0511.txt
+++ b/pep-0511.txt
@@ -12,10 +12,11 @@
Abstract
========
-Propose an API to register AST and bytecode transformers. Add also ``-o
-OPTIM_TAG`` command line option to change ``.pyc`` filenames. Raise an
-``ImportError`` exception on import if the ``.pyc`` file is missing and
-the code transformers required to transform the code are missing. code
+Propose an API to register bytecode and AST transformers. Add also ``-o
+OPTIM_TAG`` command line option to change ``.pyc`` filenames, ``-o
+noopt`` disables the peephole optimizer. Raise an ``ImportError``
+exception on import if the ``.pyc`` file is missing and the code
+transformers required to transform the code are missing. code
transformers are not needed code transformed ahead of time (loaded from
``.pyc`` files).
@@ -39,16 +40,14 @@
Usage 1: AST optimizer
----------------------
-Python 3.6 optimizes the code using a peephole optimizer. By
-definition, a peephole optimizer has a narrow view of the code and so
-can only implement basic optimizations. The optimizer rewrites the
-bytecode. It is difficult to enhance it, because it written in C.
-
Transforming an Abstract Syntax Tree (AST) is a convenient
way to implement an optimizer. It's easier to work on the AST than
working on the bytecode, AST contains more information and is more high
level.
+Since the optimization can done ahead of time, complex but slow
+optimizations can be implemented.
+
Example of optimizations which can be implemented with an AST optimizer:
* `Copy propagation
@@ -101,7 +100,9 @@
---------------------
A preprocessor can be easily implemented with an AST transformer. A
-preprocessor has various and different usages. Examples:
+preprocessor has various and different usages.
+
+Some examples:
* Remove debug code like assertions and logs to make the code faster to run
it for production.
@@ -125,10 +126,49 @@
`MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of
examples and use cases.
+This PEP does not add any new code transformer. Using a code transformer
+will require an external module and to register it manually.
+
See also `PyXfuscator <https://bitbucket.org/namn/pyxfuscator>`_: Python
obfuscator, deobfuscator, and user-assisted decompiler.
+Usage 3: Disable all optimization
+---------------------------------
+
+Ned Batchelder asked to add an option to disable the peephole optimizer
+because it makes code coverage more difficult to implement. See the
+discussion on the python-ideas mailing list: `Disable all peephole
+optimizations
+<https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_.
+
+This PEP adds a new ``-o noopt`` command line option to disable the
+peephole optimizer. In Python, it's as easy as::
+
+ sys.set_code_transformers([])
+
+It will fix the `Issue #2506 <https://bugs.python.org/issue2506>`_: Add
+mechanism to disable optimizations.
+
+
+Usage 4: Write new bytecode optimizers in Python
+------------------------------------------------
+
+Python 3.6 optimizes the code using a peephole optimizer. By
+definition, a peephole optimizer has a narrow view of the code and so
+can only implement basic optimizations. The optimizer rewrites the
+bytecode. It is difficult to enhance it, because it written in C.
+
+With this PEP, it becomes possible to implement a new bytecode optimizer
+in pure Python and experiment new optimizations.
+
+Some optimizations are easier to implement on the AST like constant
+folding, but optimizations on the bytecode are still useful. For
+example, when the AST is compiled to bytecode, useless jumps can be
+emited because the compiler is naive and does not try to optimize
+anything.
+
+
Use Cases
=========
@@ -199,17 +239,85 @@
available.
+Code transformer API
+====================
+
+A code transformer is a class with ``ast_transformer()`` and/or
+``code_transformer()`` methods (API described below) and a ``name``
+attribute.
+
+For efficiency, do not define a ``code_transformer()`` or
+``ast_transformer()`` method if it does nothing.
+
+The ``name`` attribute (``str``) must be a short string used to identify
+an optimizer. It is used to build a ``.pyc`` filename. The name must not
+contain dots (``'.'``), dashes (``'-'``) or directory separators: dots
+are used to separated fields in a ``.pyc`` filename and dashes areused
+to join code transformer names to build the optimizer tag.
+
+.. note::
+ It would be nice to pass the fully qualified name of a module in the
+ *context* when an AST transformer is used to transform a module on
+ import, but it looks like the information is not available in
+ ``PyParser_ASTFromStringObject()``.
+
+
+code_transformer()
+------------------
+
+Prototype::
+
+ def code_transformer(code, consts, names, lnotab, context):
+ ...
+ return (code, consts, names, lnotab)
+
+Parameters:
+
+* *code*: the bytecode (``bytes``)
+* *consts*: a sequence of constants
+* *names*: tuple of variable names
+* *lnotab*: table mapping instruction offsets to line numbers (``bytes``)
+
+The code transformer is run after the compilation to bytecode
+
+
+ast_transformer()
+------------------
+
+Prototype::
+
+ def ast_transformer(tree, context):
+ ...
+ return tree
+
+Parameters:
+
+* *tree*: an AST tree
+* *context*: an object with a ``filename`` attribute (``str``)
+
+It must return an AST tree. It can modify the AST tree in place, or
+create a new AST tree.
+
+The AST transformer is called after the creation of the AST by the
+parser and before the compilation to bytecode. New attributes may be
+added to *context* in the future.
+
+
Changes
=======
-This PEP proposes to add an API to register code transformers.
+In short, add:
-The transformation can done ahead of time. It allows to implement
-powerful but expensive transformations.
+* ``-o OPTIM_TAG`` command line option
+* ``ast.Constant``
+* ``ast.PyCF_TRANSFORMED_AST``
+* ``sys.get_code_transformers()``
+* ``sys.implementation.optim_tag``
+* ``sys.set_code_transformers(transformers)``
-API for code transformers
--------------------------
+API to get/set code transformers
+--------------------------------
Add new functions to register code transformers:
@@ -222,43 +330,15 @@
transformer B can give a different output than running transformer B an
then transformer A.
-API of an code transformer:
+Example to prepend a new code transformer::
-* An code transformer is a class with ``ast_transformer()`` and/or
- ``bytecode_transformer()`` methods and a ``name`` attribute. Example::
+ transformers = sys.get_code_transformers()
+ transformers.insert(0, new_cool_transformer)
+ sys.set_code_transformers(transformers)
- class MyTransformer:
- name = 'my_transformer'
-
- def bytecode_transformer(self, bytecode, ...):
- ...
- return tree
-
- def ast_transformer(self, tree, context):
- ...
- return tree
-
- ``bytecode_transformer()``: *bytecode* is bytes objects.
-
- ``ast_transformer()``: *tree* is an AST tree and *context* is an
- object with a ``filename`` attribute (``str``). New attributes may be
- added to *context* in the future. It must return an AST tree. It can
- modify the AST tree in place, or create a new AST tree.
-
-* The ``name`` attribute (``str``) must be a short string used to
- identify an optimizer. The name must not contain ``.`` (dot), ``-``
- (dash) characters or directory separators: ``.`` is used to separated
- fields in a ``.pyc`` filename and ``-`` is used to join AST
- transformer names to build the optimizer tag.
-* The AST transformer is called after the creation of the AST by the
- parser and before the compilation to bytecode
-* The bytecode transformer is after the compilation to bytecode
-
-.. note::
- It would be nice to pass the fully qualified name of a module in the
- *context* when an AST transformer is used to transform a module, but
- it looks like the information is not available in
- ``PyParser_ASTFromStringObject()``.
+All AST tranformers are run sequentially (ex: the second transformer
+gets the input of the first transformer), and then all bytecode
+transformers are run sequentially.
Optimizer tag
@@ -269,7 +349,7 @@
* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
The default optimization tag is ``'opt'``.
* Add a new ``-o OPTIM_TAG`` command line option to set
- ``sys.implementation.optim_tag``
+ ``sys.implementation.optim_tag``.
Changes on ``importlib``:
@@ -302,6 +382,19 @@
optimizer tag (``'opt'``).
+Peephole optimizer
+------------------
+
+By default, ``sys.implementation.optim_tag`` is ``opt`` and
+``sys.get_code_transformers()`` returns a list of one code transformer:
+the peephole optimizer (optimize the bytecode).
+
+Use ``-o noopt`` to disable the peephole optimizer. In this case, the
+optimizer tag is ``noopt`` and no code transformer is registered.
+
+Using the ``-o opt`` option has not effect.
+
+
AST enhancements
----------------
@@ -326,8 +419,8 @@
node type
-Example
-=======
+Examples
+========
.pyc filenames
--------------
@@ -355,10 +448,40 @@
=========================== ==================
+Bytecode transformer
+--------------------
+
+Scary bytecode transformer replacing all strings with
+``"Ni! Ni! Ni!"``::
+
+ import sys
+
+
+ class BytecodeTransformer:
+ name = "knights_who_say_ni"
+
+ def code_transformer(self, code, consts, names, lnotab, context):
+ consts = ['Ni! Ni! Ni!' if isinstance(const, str) else const
+ for const in consts]
+ return (code, consts, names, lnotab)
+
+
+ # replace existing code transformers with our bytecode transformer
+ sys.set_code_transformers([BytecodeTransformer()])
+
+ # execute code which will be transformed by ast_transformer()
+ exec("print('Hello World!')")
+
+Output::
+
+ Ni! Ni! Ni!
+
+
AST transformer
---------------
-Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
+Similary to the bytecode transformer example, the AST transformer also
+replaces all strings with ``"Ni! Ni! Ni!"``::
import ast
import sys
@@ -381,10 +504,8 @@
return tree
- # append our AST transformer after existing transformers
- transformers = sys.get_code_transformers()
- transformers.append(ASTTransformer())
- sys.set_code_transformers(transformers)
+ # replace existing code transformers with the new AST transformer
+ sys.set_code_transformers([ASTTransformer()])
# execute code which will be transformed by ast_transformer()
exec("print('Hello World!')")
@@ -397,8 +518,8 @@
Other Python implementations
============================
-The PEP 511 should be implemented be all Python implementation. The AST
-emited by the parser is not specified.
+The PEP 511 should be implemented by all Python implementation, but the
+bytecode and the AST are not standardized.
By the way, even between minor version of CPython, there are changes on
the AST API. There are differences, but only minor differences. It is
@@ -432,8 +553,13 @@
the Python semantics since no guard is used to disable optimization if
something changes.
-Issue #17515: `Add sys.setasthook() to allow to use a custom AST
-optimizer <https://bugs.python.org/issue17515>`_.
+In 2015, Victor Stinner wrote the `fatoptimizer
+<http://fatoptimizer.readthedocs.org/>`_ project, an AST optimizer
+specializing functions using guards.
+
+The Issue #17515 `"Add sys.setasthook() to allow to use a custom AST"
+optimizer <https://bugs.python.org/issue17515>`_ was a first attempt of
+API for code transformers, but specific to AST.
Python Preprocessors
@@ -448,8 +574,8 @@
preprocessor directives in Python, like ``#define`` and ``#ifdef``
-Modify the bytecode
--------------------
+Bytecode transformers
+---------------------
* `codetransformer <https://pypi.python.org/pypi/codetransformer>`_:
Bytecode transformers for CPython inspired by the ``ast`` module’s
@@ -465,10 +591,6 @@
See also:
* `BytecodeAssembler <http://pypi.python.org/pypi/BytecodeAssembler>`_
-* `Issue #2506 <https://bugs.python.org/issue2506>`_: Add mechanism to
- disable optimizations
-* `[Python-ideas] Disable all peephole optimizations
- <https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_
Copyright
--
Repository URL: https://hg.python.org/peps
More information about the Python-checkins
mailing list