Parser or regex ?

[Graham Fawcett]

Fuzzyman wrote:
> Hello all,
>
> I'm writing a module that takes user input as strings and (effectively)
> translates them to function calls with arguments and keyword
> arguments.to pass a list I use a sort of 'list constructor' - so the
> syntax looks a bit like :
>
>    checkname(arg1, "arg 2", 'arg 3', keywarg="value",
> keywarg2='value2', default=list("val1", 'val2'))
>
> Worst case anyway :-)

Since you've taken the care to use Python syntax, you could do worse
than using standard-library tools that are designed for parsing Python.

>>> import compiler
>>> from compiler import ast
>>> src = """
... checkname(arg1, "arg 2", 'arg 3', keywarg="value",
... keywarg2='value2', default=list("val1", 'val2'))
... """
>>> node = compiler.parse(src, mode='eval')
>>> node
Expression(CallFunc(Name('checkname'), [Name('arg1'), Const('arg 2'),
Const('arg 3'), Keyword('keywarg', Const('value')), Keyword('keywarg2',
Const('value2')), Keyword('default', CallFunc(Name('list'),
[Const('val1'), Const('val2')], None, None))], None, None))
>>> dir(node)
['__doc__', '__init__', '__iter__', '__module__', '__repr__', 'asList',
'getChildNodes', 'getChildren', 'node']

In this case, compiler.parse returns a compiler.ast.Expression object,
acting as the root node of an abstract syntax tree.

The compiler.visitor package may also be of help in writing tree-walker
code to extract the bits you want. With the example session above,

class KwVisitor(compiler.visitor.ASTVisitor):

    def visitKeyword(self, node):
        print '>>', node

compiler.visitor.walk(node, KwVisitor())

would output:

>> Keyword('keywarg', Const('value'))
>> Keyword('keywarg2', Const('value2'))
>> Keyword('default', CallFunc(Name('list'), [Const('val1'), Const('val2')], None, None))


Note that with a bit of black magic, you can modify the AST, and then
compile the transformed AST into a callable codeblock. Tools like
Quixote's PTL template language use this approach to introduce new
behaviours into Python-like code at compliation time.

Graham