Painful?: Using the ast module for metaprogramming

[Joseph Garvin]

I decided to try using the ast module to see how difficult or not it
was to use for metaprogramming. So I tried writing a decorator that
would perform a simple transformation of a function's code. It was
certainly not as easy as I had guessed, but I did succeed so it's not
impossible. The issues I encountered might suggest changes to the ast
module, but since this is my first time messing with it I think it's
equally likely I'm just ignorant of the best way to handle them.
Questions:

-If I have the source to a single function definition and I pass it to
ast.parse, I get back an ast.Module. Why not an ast.FunctionDef?

-An ast.Name always has an ast.Load or ast.Store "context" associated
with it. This is a bit odd, since based on the actual context (what
the parent of the ast.Name node is) it is always clear whether the
variable needs to be loaded or stored -- loaded when the node is the
child of an expression and stored when the node is the target of an
assignment. Is there some weird corner case that necessitates this
redundancy? It adds a lot of noise when you're trying to interpret
trees, and the ast.Load and ast.Store objects themselves don't seem to
contain any data.

-Why can't orelse for ast.If and ast.While default to empty []? If you
make an ast.While object by hand but don't specify that orelse is
empty, you get an error when later trying to compile it. This seems
silly since by default not specifying an else in code results in the
ast module generating a node with orelse=[].

-Why can't keywords and args for ast.Call default to empty []? Same
problem as with orelse.

-Why can't I eval a functiondef to get back a function object? As it
stands, I have to work around this by giving the functiondef a unique
name, exec'ing the AST, and then doing a lookup in locals[] to get the
resulting function object. This is a nasty hack.

-The provided NodeTransformer class is useful, but provides no way
(that I can see) to replace a single statement with multiple
statements, because multiple statements would constitute multiple
nodes. To work around this, I take the code I want to swap in, and
wrap it in a "while 1:" block with a break at the end to ensure it
only runs once and doesn't loop. Again, a kludge.

-The module provides no means to convert an AST back into source code,
which would be nice for debugging.

-It would be nice if decorators were passed a function's AST instead
of a function object. As it is I have to use inspect.getsource to
retrieve the source for the function in question, and then use
ast.parse, which is a bit inefficient because the cpython parser has
to already have done this once before. Again, it feels like a hack.
Making decorators take AST's would obviously be a compatibility
breaking change, since you'd then have to compile them before
returning, so alternatively you could have "ast decorators" that would
use a different prefix symbol in place of @, or you could change it so
that decorators got passed the AST but the AST had a __call__ method
that would cause the AST to parse itself and become the resulting
function object in place and then execute itself, so until the first
time it was called it was an AST but after that was a function object
(it would 'lazily' become a function). I think that wouldn't break
most code.

I have some ideas for what an easier API would look like but I want to
be sure these are real issues first and not just me doing it wrong ;)

Regards,

Joe