[Python-ideas] History on proposals for Macros?

[Steven D'Aprano]

On Sat, Mar 28, 2015 at 12:50:09PM -0700, Andrew Barnert wrote:
> On Mar 28, 2015, at 10:26, Steven D'Aprano <steve at pearwood.info> wrote:
> > 
> >> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote:
> >> [...]
> >> The goal is to create things that look like functions but have access to
> >> the expression that was passed in.
> > 
> >>    assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
> >> assertRaises function, not before
> > 
> >> Generally one constructs something that looks like a function but, rather
> >> than receiving a pre-evaluated input, receives a syntax tree along with the
> >> associated context.  This allows that function-like-thing to manipulate the
> >> expression and to control the context in which the evaluation occurs.
> > 
> > How will the Python compiler determine that assertRaises should receive 
> > the syntax tree rather than the evaluated 1/0 expression (which of 
> > course will raise)? The information that assertRaises is a "macro" is 
> > not available at compile time.
> 
> Well, it _could_ be available. At the time you're compiling a scope (a 
> function, class, or top-level module code), if it uses an identifier 
> that's the name of a macro in scope, the compiler expands the macro 
> instead of compiling in a function call.

Perhaps I trimmed out too much of Matthew's comment, but he did say he 
isn't talking about C-style preprocessor macros, so I think that if you 
are imagining "expanding the macro" C-style, you're barking up the wrong 
tree. From his description, I don't think Lisp macros are quite the 
right description either. (As always, I welcome correction if I'm the 
one who is mistaken.)

C macros are more or less equivalent to a source-code rewriter: if you 
define a macro for "foo", whenever the compiler sees a token "foo", it 
replaces it with the body of the macro. More or less.

Lisp macros are different, and more powerful:

http://c2.com/cgi/wiki?LispMacro
http://cl-cookbook.sourceforge.net/macros.html

but I don't think that's what Matthew wants either. The critical phrase 
is, I think:

"one constructs something that looks like a function but, rather than 
receiving a pre-evaluated input, receives a syntax tree along with the 
associated context"

which I interpret in this way:

Suppose we have this chunk of code creating then using a macro:

    macro mymacro(expr):
        # process expr somehow

    mymacro(x + 1)


That is *roughly* equivalent (ignoring the part about context) to what 
we can do today:

    def myfunction(expr):
        assert isinstance(expr, ast.Expression)
        # process expr somehow

    tree = ast.parse('x + 1', mode='eval')
    myfunction(tree)


The critical difference being that instead of the author writing code to 
manually generate the syntax tree from a string at runtime, the compiler 
automatically generates the tree from the source code at compile time.

This is why I think that it can't be done by Python. What should the 
compiler do here?

callables = [myfunction, mymacro]
random.shuffle(callables)
for f in callables:
    f(x + 1)


If that strikes you as too artificial, how about a simpler case?

from mymodule import name
name(x + 1)


If `name` will refer to a function at runtime, the compiler needs to 
generate code which evaluates x+1 and passes the result to `name`; but 
if `name` will refer to a macro, the compiler needs to generate an ast 
(plus context) and pass it without evaluating it. To do that, it needs 
to know at compile-time which objects are functions and which are 
macros, but that is not available until runtime.

But we might be able to rescue this proposal by dropping the requirement 
that the compiler knows when to pass the syntax tree and when to 
evaluate it. Suppose instead we had a lightweight syntax for generating 
the AST plus grabbing the current context:

    x = 23
    spam(x + 1, !(x+1))  #  macro syntax !( ... )


Now the programmer is responsible for deciding when to use an AST and 
when to evaluate it, not the compiler, and "macros" become regular 
functions which just happen to expect an AST as their argument.


[...]
> If such a light-lambda syntax reduced the desire for macros down to 
> the point where it could be ignored, and if that desire weren't 
> _already_ low enough that it can be ignored, it would be worth adding. 
> I think the second "if" is where it fails, not the first, but I could 
> be wrong.

I presume that Matthew wants the opportunity to post-process the AST, 
not merely evaluate it. If all you want is to wrap some code an an 
environment in a bundle for later evaluation, you are right, a function 
will do the job. But it's hard to manipulate byte code, hence the desire 
for a syntax tree.


-- 
Steve