Partial Parsing of Python

[Peter Eckersley]

Hi,

I'm currently working on a python-based alternative to a standard *nix
shell.  I imagine lots of other people have built such animals, but this
one is supposed to be graphical and include nifty features like smart
auto-completion and running commentary on one's commands (which should be
handy for newbies).

Anyway, having implemented some of the completion features the simple 
way-- using tokenize.tokenize(), string analysis, and the odd call to
parser.expr() to guess what the user might want to type next, I've
reached the conclusion that doing too much more will lead to Madness
and Spaghetti Code.

So the alternative is of course to use a proper python parser, which 
can include hooks suggesting how bits of the grammar can be completed.
Basically, what this parser needs to do is attempt to accept its input,
noting along the way when it meets the user's cursor.  If it fails,
it needs to preserve an incomplete parse tree, record which production
rule it was trying to use, and which symbol in that rule caused the
problem.

It's not immediately obvious how to achieve this with a minimum of fuss.
A few different strategies might be:

* hack the cPython parser to provide this extra stuff (not having emersed
  myself in the parser, I imagine this could be a lot of work, but would
  love to hear information to the contrary, or receive few pointers as
  to how it might be done)

* use bison to implement a new parser from scratch (ugly,
  perilous, and I'd have to learn yacc/bison along the way :( )

* write a minimalist, brute-force recursive descent parser in python,
  which would be a more-or-less direct translation of the grammar, but
  would almost certainly be damnably slow

Suggestions about which strategy to pursue would be gratefully
appreciated!

(CCs off-list, IINTMT)

-- 
Peter Eckersley
Department of Computer Science   &                  mailto:pde at cs.mu.oz.au 
IP Research Institute of Australia             http://www.cs.mu.oz.au/~pde
The University of Melbourne