[Python-ideas] PEP 511: API for code transformers

[Brett Cannon]

On Sat, 16 Jan 2016 at 19:38 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 17 January 2016 at 04:28, Brett Cannon <brett at python.org> wrote:
> >
> > On Fri, 15 Jan 2016 at 09:40 Victor Stinner <victor.stinner at gmail.com>
> > wrote:
> >>
> >> 2016-01-15 18:22 GMT+01:00 Brett Cannon <brett at python.org>:
> >> > I just wanted to point out to people that the key part of this PEP is
> >> > the
> >> > change in semantics of `-O` accepting an argument.
> >>
> >> The be exact, it's a new "-o arg" option, it's different from -O and
> >> -OO (uppercase). Since I don't know what to do with -O and -OO, I
> >> simply kept them :-D
> >>
> >> > I should also point out that this does get tricky in terms of how to
> >> > handle
> >> > the stdlib if you have not pre-compiled it, e.g., if the first module
> >> > imported by Python is the encodings module then how to make sure the
> AST
> >> > optimizers are ready to go by the time that import happens?
> >>
> >> Since importlib reads sys.implementation.optim_tag at each import, it
> >> works fine.
> >>
> >> For example, you start with "opt" optimizer tag. You import everything
> >> needed for fatoptimizer. Then calling sys.set_code_transformers() will
> >> set a new optimizer flag (ex: "fat-opt"). But it works since the
> >> required code transformers are now available.
> >
> >
> > I understand all of that; my point is what if you don't compile the
> stdlib
> > for your optimization? You have to import over 20 modules before user
> code
> > gets imported. My question is how do you expect the situation to be
> handled
> > where you didn't optimize the stdlib since the 'encodings' module is
> > imported before anything else? If you set your `-o` flag and you want to
> > fail imports if the .pyc isn't there, then wouldn't that mean you are
> going
> > to fail immediately when you try and import 'encodings' in
> Py_Initialize()?
>
> I don't think that's a major problem - it seems to me that it's the
> same as going for "pyc only" deployment with an embedded Python
> interpreter, and then forgetting to a precompiled standard library in
> addition to your own components. Yes, it's going to fail, but the bug
> is in the build process for your deployment artifacts rather than in
> the runtime behaviour of CPython.
>

It is the same, and that's my point. If we are going to enforce this import
requirement of having a matching .pyc file in order to do a proper import,
then we are already requiring an offline compilation which makes the
dynamic registering of optimizers a lot less necessary.

Now if we tweak the proposed semantics of `-o` to say "import these of kind
of optimized .pyc file *if you can*, otherwise don't worry about it", then
having registered optimizers makes more sense as that gets around the
bootstrap problem with the stdlib. This would require optimizations to be
module-level and not application-level, though. This also makes the
difference between `-o` and `-O` even more prevalent as the latter is not
only required, but then restricted to only optimizations which affect what
syntax is executed instead of what AST transformations were applied. This
also means that the file name of the .pyc files should keep `opt-1`, etc.
and the AST transformation names get appended on as it would stack `-O` and
`-o`.

It really depends on what kinds of optimizations we expect people to do. If
we expect application-level optimizations then we need to enforce universal
importing of bytecode because it may make assumptions about other modules.
But if we limit it to module-level optimizations then it isn't quite so
critical that the .pyc files pre-exist, making it such that `-o` can be
more of a request than a requirement for importing modules of a certain
optimization. That also means if the same AST optimizers are not installed
it's no big deal since you just work with what you have (although you could
set it to raise an ImportWarning when an import didn't find a .pyc file of
the requested optimization *and* the needed AST optimizers weren't
available either).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160118/360443c9/attachment-0001.html>