Getting in to metaprogramming

[Steven D'Aprano]

On Fri, 28 Nov 2008 05:05:10 +0200, Hendrik van Rooyen wrote:

> "Terry Reedy" <tjreedy at udel.edu> wrote:
> 
> 
>> Hendrik van Rooyen wrote:
>>
>> > I am using the term in the restricted sense of Python writing Python
>> > source.
>> >
>> > Given that, can anybody think of an example that you could not do
>> > with a class?  (excepting the "stored procedure" aspect)
>>
>> I am not sure I understand your question.
>>
>> def iterize(recursive_function_text):
>>      <code to parse input and fill a template> return
>>      equivalent_iterative_function_text
>>
>> where input and output are both Python code.  If one were to implement
>> this by compiling the input to AST form and then walking the tree, the
>> AST node classes would be involved, but I would scarely say the
>> translation was done by the classes, as opposed to functions which
>> might or might not be attacked to a class as methods.
>>
>>
> I am not sure I understand the answer - if the input and output are both
> bits of python source, then the output must be stored and evaluated or
> imported to get it executed, right?

I think there's some confusion here. Python, like all general-purpose 
programming languages, is able to process text. Python source code, like 
that of almost all programming languages, is text.

Therefore Python can process Python source code as text: it can read it 
and process it, or it can write it. There's nothing mysterious about 
this, although how useful it is depends on the nature of the processing.

Once you have that Python source code, it is no different whether you 
wrote it by hand, or wrote it with the aid of another program. To execute 
it, you have to execute it.

Here's a trivial example of metaprogramming. Suppose I am interested in 
the compilation time of Python code, rather than the execution time. I 
might do this:

numitems = 100000
template = """def main():
    # %s
    %s%s
    return L
"""

def make_file(name, comment, setup, body):
    f = open(name, 'w')
    f.write(template % (comment, setup, body))
    f.close()

body = 'L.append(None)\n'
make_file('dumb.py', 'Create a big list the dumb way.', 
    'L = []\n', body*numitems)
make_file('smart.py', 'Create a big list the smart way.', 
    '', 'L = [None]*%d\n' % numitems)



At this point, I have two source files which I can do anything I like 
with. It would have been a real pain to have written the first one by 
hand, even with a clever editor. I can post-process them, run them 
through other tools, even open them up in an editor and manipulate them 
by hand (although that defeats the purpose of metaprogramming). Now I can 
run the source files as input to another piece of code, and get a (very 
rough) estimate of the compilation time:

from time import time
start = time()
execfile('dumb.py')  # bypass the import mechanism
tdumb = time() - start
print "Time taken to compile dumb.py is %f seconds" % tdumb

start = time()
execfile('smart.py')
tsmart = time() - start
print "Time taken to compile smart.py is %f seconds" % tsmart



On my computer, I get:

Time taken to compile dumb.py is 2.236122 seconds
Time taken to compile smart.py is 0.003754 seconds



-- 
Steven