Partial Parsing of Python

[holger krekel]

Hi Peter,

You wrote:
> I'm currently working on a python-based alternative to a standard *nix
> shell.  I imagine lots of other people have built such animals, but this
> one is supposed to be graphical and include nifty features like smart
> auto-completion and running commentary on one's commands (which should be
> handy for newbies).

That's great! I can offer you code which will help you. 
Moreover some of us had good discussions at EuroPython about *exactly*
this type of project. For a start look at my alpha-version of a new 
command line completer:

    http://home.trillke.net/~hpk/rlcompleter2.py

Michael Hudson moreover has done nice work with replacing readline
with curses. 

    http://starship.python.net/crew/mwh/hacks/pyrepl.html

> Anyway, having implemented some of the completion features the simple 
> way-- using tokenize.tokenize(), string analysis, and the odd call to
> parser.expr() to guess what the user might want to type next, I've
> reached the conclusion that doing too much more will lead to Madness
> and Spaghetti Code.

completly correct. 

> So the alternative is of course to use a proper python parser, which 
> can include hooks suggesting how bits of the grammar can be completed.
> Basically, what this parser needs to do is attempt to accept its input,
> noting along the way when it meets the user's cursor.  If it fails,
> it needs to preserve an incomplete parse tree, record which production
> rule it was trying to use, and which symbol in that rule caused the
> problem.

Actually the problem goes deeper. Have a look at my TryParser class
in rlcompleter2.py for a start. It does handle many cases and should
give you some more ideas :-)

Actually the REAL problem to solve first is to have Multi-Line editing. 
Then all sorts of partial parsers *and* partial evaluation.  Python is
just too damn dynamic to have that one easily.

So what to do? Our idea at EuroPython was to use PyGame (the wrapper
around SDL) to get rid of the uglyness and restrictiveness of 
readline and curses alltogether.  We want to have a REAL playground 
and concentrate on our ideas.

Once i slow down from the mind-blowing EuroPython discussions and talks
i check how easy that is.

Drop me a note if you are interested to stay tuned.

    holger