PyLR -- Fast LR parsing in python

What is PyLR?
What is the current state of PyLR?
Where do I get PyLR?
What will be added to PyLR?
Where do I find out about parsing theory?
How can I contribute to PyLR?

What is PyLR?

PyLR is a package of tools for creating efficient parsers in python, commonly known as a compiler compiler. PyLR is currently under development. A ful release is almost complete, but there are still a few missing features that would make it much nicer.

PyLR (pronounced 'pillar') was motivated by the frequencly with which parsers are hand coded in python, the performance demands that these parsers are subject to (you just can't beat native machine code for speed...), and academic curiosity (I wanted to really know how LR parsing works).

What is the current state of PyLR?

PyLR currently has class interfaces to a Grammar, a Lexer, an extension module defining a parsing engine builtin type, and a parser generator script. All of these components are based on sound parsing theory, but nevertheless haven't been tested by anyone but it's author. The code as is stands can definitely be of use to anyone hand writing a parser in python, but some of the nicer things in the complete package just haven't been done yet .

PyLR is therefore under development, as it will always be. PyLR will be given a release number once it supplies the following tools:

write an 'engine' module that implements the LR parsing algorythm in C with callbacks to python functions. (done)
write a Lexer class using re (done)
write a Grammar class that will take as input a context free grammar and produce the parsing tables necessary to complement the engine. This is to be done with LR(1) grammars (done and then deleted -- extremely inefficient) and LALR(1) Grammars(done, except with epsilon (empty) productions, much more efficient).
add a user interface -- manually write a lexer and Grammar using the exisiting classes to parse lexer and grammar specifications modelled after lex/flex and yacc/bison. (done for Grammars)
write documentation. (usable, but not done)
(post release) add grammars to various languages to the distribution.

In addtion, I have the following plan for the project:

make 'epsilon' (empty) productions work (many of them work now, but not all)
optimize the Lexer. Try to join it into one regular expression and derive function calls from match object data. (done, still the slowest part of parsing)
add error specification routines.
change the parser generation algorithm to use only kernel LALR(1) items in the computation of shift actions and gotos in the goto table. This should significantly enhance the rate of parser generation, which is currently a bit slow, but certainly acceptable for medium-sized grammars (< ~100 productions) (done!) this version
write a Parser for sql, as used in gadfly
add operator precedence as an option to the parser specification (further down the road...)

These things will probably be done over the next month or two (as I only have free time to give to this project...Ahemmm...).

Where do I get PyLR?

You can get PyLR in one of two places, here or here. Both versions will be in sync with each other.

What will be added to PyLR?

In addition to the list of things to finish before a full release, is published, PyLR could be used as the basis for an efficient datapath analyzer (optimizer), for a front end to translation from one language to another, for type checking code, etc.

As soon as the first release is completed, Tools to aid in all these things could well be added to the package. Also, anyone wanting to contribute parser specifications for languages of general use is most welcome.

Where do I find out more about parsing?

Parsing was for a long time a big challenge for computer scientists. The need for computer parsing originally came about with the first writing of compilers. Since then, the theory behind parsing has been studied in depth and has pretty much stabilized as it no longer really presents a big problem in terms of speed or size in terms of parsing todays computer languages. One standard means of parsing that has been used for years because of its efficiency is LR parsing (more particularly, LALR parsing). A lot of good information is in Lex and Yacc , The Dragon Book , and it seems like the only place to find good info on LALR parsing is in

DeRemer, F.; and Pennello, T.Efficient computation of LALR(1) look-ahead sets, ACM Trans.
 Program. Lang. Syst. 4 (1982), 615-649.

Finally, to find out how to use PyLR, see thePyLR manual

How do I contribute to PyLR?

mail me.