[Python-ideas] PEP 511: API for code transformers

[Victor Stinner]

2016-01-15 18:22 GMT+01:00 Brett Cannon <brett at python.org>:
> I just wanted to point out to people that the key part of this PEP is the
> change in semantics of `-O` accepting an argument.

The be exact, it's a new "-o arg" option, it's different from -O and
-OO (uppercase). Since I don't know what to do with -O and -OO, I
simply kept them :-D

> I should also point out that this does get tricky in terms of how to handle
> the stdlib if you have not pre-compiled it, e.g., if the first module
> imported by Python is the encodings module then how to make sure the AST
> optimizers are ready to go by the time that import happens?

Since importlib reads sys.implementation.optim_tag at each import, it
works fine.

For example, you start with "opt" optimizer tag. You import everything
needed for fatoptimizer. Then calling sys.set_code_transformers() will
set a new optimizer flag (ex: "fat-opt"). But it works since the
required code transformers are now available.

The tricky part is more when you want to deploy an application without
the code transformer, you have to ensure that all .py files are
compiled to .pyc. But there is no technical issues to compile them,
it's more a practical issue.

See my second email with a lot of commands, I showed how .pyc are
created with different .pyc filenames. Or follow my commands to try my
"fatpython" fork to play yourself with the code ;-)

> And lastly, Victor proposes that all .pyc files get an optimization tag.
> While there is nothing technically wrong with that, PEP 488 purposefully
> didn't do that in the default case for backwards-compatibility, so that will
> need to be at least mentioned in the PEP.

The PEP already contains:
https://www.python.org/dev/peps/pep-0511/#optimizer-tag
"Remove also the special case for the optimizer level 0 with the
default optimizer tag 'opt' to simplify the code."

Code relying on the exact .pyc filename (like unit tests) already have
to be modified to use the optimizer tag. It's just an opportunity to
simplify the code. I don't really care of this specific change ;-)

Victor