[Python-Dev] Release of astoptimizer 0.3

[Victor Stinner]

>> Projects using the same code base for Python 2 and Python 3 contain a
>> lot of inefficient code. For example, using the six library, a simple
>> Unicode literal strings becomes a function code: u('unicode').
>
> But are you able to do enough static analysis to feel comfortable that
> this is the right u() function? IIRC you said earlier that you're not
> even capable of recognizing "len = ord; print(len('a'))" -- if that is
> really true, I'm very worried about your optimizer's capacity for
> breaking code. I'm not talking about "behind-their-back" changes to
> __builtins__ or patching of the module globals. I'm talking about
> detecting straightforward definitions that override the identifiers
> you are replacing.

astoptimizer is still experimental, but I prefer to release early and
release often, because I already get interesting feedback.

"from math import pow as len" is already supported in the version 0.3,
and "len=ord" was fixed in the version 0.3.1.

astoptimizer is supposed to handle any instruction setting variables
(if I forgot to handle a specific instruction, it's a bug). My initial
goal was to optimize "x=1; return x", but then I realized that I must
handle take care of all variables, because they may shadow builtin
functions or constants.

The following AST statements creates a "new" namespace (inherit some
properties from the parent):
 - Module
 - FunctionDef
 - ClassDef

The following AST statements set variables or have an impact on scope:
 - Assign, AugAssign, Del
 - Global, Nonlocal
 - Import, ImportFrom
 - With
 - arguments (of FunctionDef)
 - comprehension (of ListComp or GeneratorExp)

There is an experimental support for assignments. If an unsupported
assignment is found (ex: ((x, y), z) = x_y_z), all optimizations on
names (ex: len("abc") or math.e) are disabled. For example, "from re
import *" disables optimizations (but "from math import *" is
supported).

>> I expect that astoptimizer will be able to remove (or at least
>> reduce!) the overhead of the six library and all checks on the Python
>> version ("if PYTHON3: ... else: ...").
>
> Hm. Wouldn't it be just as easy to run a source-to-source translator
> to remove six artefacts instead of an ast optimizer?

You mean something like 2to3? I understood that the six module is
written for developers who prefer to use the same code base for Python
2 and Python 3.

With Python 3.3, if astoptimizer hooks the compile() builtin, the
optimization is enabled transparently when importing a module (during
.py => .pyc compiling). There is no need to explicitly "compile" an
application.

> Surely some convention could be adopted that is easy to use,
> and the tool to do the translation could be a lot simpler
> than an ast optimizer.

I like AST because I don't need to write my own parser.

> Sorry for being skeptical, but I'm not excited about advertising this
> as a general optimization tool unless you can make it a lot safer.

It is safer at every release. I will use the version number 1.0 when
the optimizer will be completly safe :-) (So it's only 31% safe yet!)

Victor