[pypy-dev] Re: Notes on compiler package

[Michael Hudson]

logistix <logistix at zworg.com> writes:

> I'm finally starting to get my head wrapped around the compiler package,
> and thought I'd pass some notes on to the group since it's a beast:

As the person who did some fixing up so it worked in 2.3a1, I second
that.

> 1) Other than the original AST generation, it looks like it's all pure
> python (not tying into the C source).  The original AST comes from the
> parser module, which is a pure wrapper module (although I've already
> posted python code to build ast's here)

Yep.

> 2) Internal CPython ASTs are transformed into "compiler"'s own ASTs. 
> These nodes are smarter than the internal ones; they attach appropriate
> info to "attributes".  CPython AST nodes just have a list of children,
> this module identifies and breaks out important data (for example, the
> arguments in a function def become an "args" attribute instead of
> guessing it's the first child)

Yep.  This is compiler.transformer.  It has to be said, if you were
writing a parser for Python in Python, I'd shoot for generating the
output of the transformer module rather than aping what the parser
module produces.

> 3) The CodeGenerators walk the ASTs and emit code into a code object.

Yup.

> 4) Speed seems reasonable (which makes me think I'm missing some calls
> into the C side of things)

You obviously haven't tried to compile anything big with it yet.  It's
slow.

> 4) The generated bytecode looks reasonably good and execs fine, but
> there are still some diffences from what CPython generates.  Here's what
> I've seen so far:
>     SET_LINENO doesn't always work the same (this is how python knows
> what line in a file threw an exception)

Hey, in 2.3 SET_LINENO doesn't even exist any more :-)

>     Doesn't have one of the few optimizations the CPython compiler does
> have.  CPython will throw out misplaced "docstrings"... strings in the
> source that aren't assigned to anything.  This throws off array indexes
> in the code objects co_const attribute.

Yeah, indices into co_consts are different (and not just in the case
you mention above, I think).  But it works -- you can compile the
stdlib with in and run the test suite happily.

> In general, the compiler package is in alot better shape than I expected.

What were you expecting?

Cheers,
M.