[Python-ideas] PEP 511: API for code transformers

[Brett Cannon]

On Fri, 15 Jan 2016 at 08:11 Victor Stinner <victor.stinner at gmail.com>
wrote:

> [SNIP]
>


> Optimizer tag
> -------------
>
> Changes:
>
> * Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
>   The default optimization tag is ``'opt'``.
> * Add a new ``-o OPTIM_TAG`` command line option to set
>   ``sys.implementation.optim_tag``.
>
> Changes on ``importlib``:
>
> * ``importlib`` uses ``sys.implementation.optim_tag`` to build the
>   ``.pyc`` filename to importing modules, instead of always using
>   ``opt``. Remove also the special case for the optimizer level ``0``
>   with the default optimizer tag ``'opt'`` to simplify the code.
> * When loading a module, if the ``.pyc`` file is missing but the ``.py``
>   is available, the ``.py`` is only used if code optimizers have the
>   same optimizer tag than the current tag, otherwise an ``ImportError``
>   exception is raised.
>
> Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can
> be compiled to import a module::
>
>     def transformers_tag():
>         transformers = sys.get_code_transformers()
>         if not transformers:
>             return 'noopt'
>         return '-'.join(transformer.name
>                         for transformer in transformers)
>
>     def use_py():
>         return (transformers_tag() == sys.implementation.optim_tag)
>
> The order of ``sys.get_code_transformers()`` matter. For example, the
> ``fat`` transformer followed by the ``pythran`` transformer gives the
> optimizer tag ``fat-pythran``.
>
> The behaviour of the ``importlib`` module is unchanged with the default
> optimizer tag (``'opt'``).
>

I just wanted to point out to people that the key part of this PEP is the
change in semantics of `-O` accepting an argument. Without this change
there is no way to cause import to pick up on optimized .pyc files that you
want it to use without abusing pre-existing .pyc filenames.

This also means that everything else is optional. That doesn't mean it
shouldn't be considered, mind you, as it makes using AST and bytecode
transformers more practical. But some `-O` change that allows user-defined
optimization tags is needed for any of this to work reasonably. From there
it's theoretically possible for someone to write their own compileall that
pre-compiles all Python code to .pyc files with a specific optimization tag
which they specify with `-O` using their own AST and bytecode transformers
and hence not need the transformation features built into sys/import.

I should also point out that this does get tricky in terms of how to handle
the stdlib if you have not pre-compiled it, e.g., if the first module
imported by Python is the encodings module then how to make sure the AST
optimizers are ready to go by the time that import happens?

And lastly, Victor proposes that all .pyc files get an optimization tag.
While there is nothing technically wrong with that, PEP 488
<https://www.python.org/dev/peps/pep-0488/> purposefully didn't do that in
the default case for backwards-compatibility, so that will need to be at
least mentioned in the PEP.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160115/03ed2f40/attachment.html>