code is data

[Kay Schluehr]

Anton Vredegoor wrote:
> Paul Boddie wrote:
>
> > Anton Vredegoor wrote:
>
> >> Yes, but also what some other posters mentioned, making Pythons internal
> >> parsing tree available to other programs (and to Python itself) by using
> >> a widely used standard like XML as its datatype.
> >
> > http://pysch.sourceforge.net/ast.html
>
> Very interesting, it led me to some sxml describing pages and it also
> tricked me into reading some Python documentation that I had always
> considered to be hiding some arcane deep Python magic. I guess now that
> Python is officially entering tree territory (as opposed to providing
> third party functionality) it seems unavoidable that Python's officially
> endorsed tree datatype will also be used for some of its internal
> structures, thereby making it more accessible to programmers like me and
> to outside programmers.

I had the same initial impression with the "deep" aspects of Python
parser API but I learned soon that I just stumbled about an awkward
kind parse-tree representation. The structures are as simple as they
could be - due to the nice LL(1) Python grammar which are reflected.

I think node manipulation is just a first basic step for defining
source transformers i.e. they provide a sound basis. For usability
purposes a template language is a far better solution.

Here is my own attempt. Defining a repeat statement ( for demonstration
purposes ) using EasyExtend requires following translation:

repeat:
    <suite>
until:
    <test>

==>

while 1:
   <suite>
   if <test>:
       break

The placeholders <suite> and <test> correspond to nodes in the
parse-tree ( and constants in symbol.py ). The repeat-stmt is defined
by an added grammar rule. The corresponding node in the parse-tree is
dispatched against Transformer.handle_repeat_stmt() during parse-tree
traversal. Matching against <suite> and <test> in the repeat_stmt node
is done by applying:

        suite_node = find_node(repeat_node, symbol.suite)
        test_node = find_node(repeat_node, symbol.test, level=1)

What becomes tedious is the creation of the corresponding while_stmt
node which is the expected result of the transformation. I actually
want to use the declaration as it is written above and insert the
suite_node and the test_node where the placeholders <suite> and <test>
are defined. Using EasyExtend one can define an extension language
where the placeholders ( and some enhanced versions of those ) become
language elements! Given this one can put everything together:

import macro    # module that corresonds to the macro extension
language

class FastTransformer(Transformer):
    def handle_repeat_stmt(self, repeat_node):
        suite_node = find_node(repeat_node, symbol.suite)
        test_node  = find_node(repeat_node, symbol.test, level=1)
        target_stmt = """
        while 1:
            <suite>
            if <test>:
                break
        """
        while_node =  macro.expand(target_stmt,
                                                  {'suite': suite_node,
'test': test_node},
                                                   goal =
symbol.while_stmt)
        return any_stmt(while_node)

This is pretty safe and works recursively: if the <suite> node contains
another <repeat_stmt> node it will be replaced as well.