ANN compiler2 : Produce bytecode from Python 2.5 Abstract Syntax Trees

[Michael Spencer]

Paul Boddie wrote:
> Martin v. Löwis wrote:
...The compiler package is largely unmaintained and
>> was known to be broken (and perhaps still is).
> 
> I don't agree entirely with the "broken" assessment. Although I'm not
> chasing the latest language constructs, the AST construction part of
> the package seems good enough to me, and apparent bugs like duplicate
> parameters in function signatures are actually documented shortcomings
> of the functionality provided.

The existing package is only lightly tested, so it's hard to say whether it's
broken or not.  The main test from test_compiler says

     def testCompileLibrary(self):
         # A simple but large test.  Compile all the code in the
         # standard library and its test suite.  This doesn't verify
         # that any of the code is correct, merely the compiler is able
         # to generate some kind of code for it.

  I certainly don't like the level of code
> documentation; from the baroque compiler.visitor, for example:
> 
> # XXX should probably rename ASTVisitor to ASTWalker
> # XXX can it be made even more generic?
> 
> However, a cursory scan of the bugs filed against the compiler module,
> trying hard to filter out other compiler-related things, reveals that
> most of the complaints are related to code generation, and the
> compiler2 module appears to be squarely aimed at this domain.

That's right, compiler2 just uses the builtin compile to get the AST, then does
the code generation in Python.  (ASTVisitor has disappeared in the process).

> 
> I find the compiler package useful - at least the bits not related to
> code generation

I think you're saying that you find the AST itself valuable.  I agree, and I've 
promoted that sort of use on this list e.g., 
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/364469

However, 2.5 ASTs are better for this purpose: they are more accurate, faster to 
create, have a more consistent node structure, and are somewhat easier to 
traverse.  I think the only reason to continue using 2.4 ASTs is for backward 
compatibility.  Moreover, the 2.5 trees will be automatically maintained as part 
of the core compile process.  Who knows what will happen to the 2.4 version?

 >... I'd be unhappy to see it dropped.
Sooner or later, I think we should drop the 2.4 AST format - it's confusing to 
have two formats, and produces unnecesary maintenance work.  I think new 
AST-manipulating apps would be better done using the 2.5 AST.

Until now, it hasn't been possible for Python apps to compile 2.5 ASTs to 
bytecode, but compiler2 is working on fixing that.

It would be straightforward to write a new transformer that took 2.5 ASTs and 
turned them into 2.4 ASTs, (and possible, but a bit harder to go the other way, 
I suspect).  But I'd rather just leave compiler alone, and document the changes 
required to use 2.5 trees.  The compiler2 package does this (see 
http://svn.brownspencer.com/pycompiler/branches/new_ast/test/pyast.py ) - and 
the changes required to a 2.4 application are easy.

Regards

Michael