Python parser that records source ranges

[logistix at cathoderaymission.net]

Jonathan Edwards <edwards at nospam.lcs.mit.edu> wrote in message news:<3F7A3AA7.9090804 at nospam.lcs.mit.edu>...
> So the basic idea is to match up the leaves of the AST with the list of 
> tokens from tokenizer, which do contain location info. I had thought of 
> that, but was hoping there was a more informative parser out there.
> Thanks.
> 
> Jonathan
> 
> 


Its really not that bad.  The more I think about it, the code
reference I sent you is way overcomplicated.  General pseudocode for
walking asts generated via parser.ast2tuple(parser.suite(code)) is:

def walk_node(node):
    if len(node) == 2 and type(node[1]) is not tuple:
        walk_token(node)
    else:
        return walk_symbol(node)

def walk_symbol(node):
    symbol_type = node[0]
    symbol_leaves = node[1:]
    for leave in symbol_leaves:
        walk_node(nod)

def walk_token(node):
    token_type = node[0]
    token_value = node[1]