[pypy-dev] Notes on compiler package

[logistix]

Christian Tismer <tismer at tismer.com> wrote: 
> 
> logistix wrote:
> > I'm finally starting to get my head wrapped around the compiler package,
> > and thought I'd pass some notes on to the group since it's a beast:
> > 
> > 1) Other than the original AST generation, it looks like it's all pure
> > python (not tying into the C source).  The original AST comes from the
> > parser module, which is a pure wrapper module (although I've already
> > posted python code to build ast's here)
> 
> Oh, is that true?
> Last time, when I analysed what it would take to
> build a compiler for Python in Python, I ended
> up in the parser module being called, finally,
> which called back into the internals stuff.
> This was for Python 2.2.2; maybe this changed now?
> 
> ciao - chris
> 
> 

Here's the recursive decent parser I wrote to build ASTs about six
months ago for 2.2 (just tested on 2.3a):

http://members.bellatlantic.net/~olsongt/concrete.zip

I haven't written an extensive test-suite yet (I was just screwing
around when I originally wrote it), but it did successfully build a tree
that could get my version of the old nines problem to compile.

It definitely needs a some rewriting (stack hog from hell), but it's a
proof of concept.  It should replace the parser module functionality.
Then if you tie that into compiler package, it looks like pure python
compilation to bytecode!  I was suprised to see how much the compiler
module did when I finally looked at it.

To test, unzip the files somewhere, adjust the hardcoded reference to
"c:\python22\lib\site-packages\concrete\misc\nines.py" as appropriate,
and run "python llparser.py".  You'll see the results of several trees I
build getting compiled and exec'ed (using the CPython compiler)

I too have trouble believing it could be this easy.   A quick grep of
compiler shows that it only uses parser to create ast Tuples in the
early phases of generation.  It doesn't use the astcompile functions on
the back end.  If anyone more familiar with the compiler internals would
like to chime in on binary dependancies I'd appreciate it.

It's getting a little late tonight, but tomorrow I'll just delete the
parser pyd from my 2.3 install and see if I can plug my parser into
compiler.

I'm also noticing some more minor problems with the compiler bytecode
generation, but the more I look the more I think the framework has the
first 90% of the work done.

-logistix

P.S. If anyone can point me to some sort of definition of what the
"Extended" in ELL(1) stands for I'd appreciate it.  Python grammar is
not LL(1) as written.

---------------------------------------
Get your free e-mail address @zworg.com