macro FAQ

[Jacek Generowicz]

dtolton at yahoo.com (Doug Tolton) writes:

> Also note that the examples presented are purposefully trivial and
> could in many instances be accomplished other ways,

I hoped that the examples that I presented satisfied the requirements
that

a) They be simple enough to understand

b) not be (at least easily) implementable using existing Python
   features

c) Not be completely useless

Did I fail ? :-)

> Essentially Macros allow you to pass in some variables that are then
> expanded into valid Python syntax which is then executed.

Well, not just variables, data in general, some of which might be
variables, some of which might be variable names (identifiers) and
some of which might be (peudo-) code.

> Question: Can't I accomplish this same functionality via string
> parsing and eval?
> 
> Answer: Eval will accept any valid Python expression and execute it. 
> This is however regarded by many as a security hole.  Macros allow
> this type of expressiveness while limiting many of the security risks.
> With a Macro you can more specifically control what types of actions
> are permissible.

I don't think that this is relevant (or true, for that matter). Lisp
macros do not expand arbitrary data that a user passes in, they appear
as literals in your source code. The security issues are the same as
those surrounding "os.system('rm -rf /')" appearing literally in your
source code.

What is relevant, is the pain involved in doing it via strings. The
(pseudo) code that you pass to a Lisp macro, not only looks exactly
like Lisp code, but it is automatically tokenized and parsed for you,
and very easily manipulatable.

> # Note, Jacek's method of using indentation is an interesting
> alternative to calling them as a
> # function.

It's probably worth pointing out, again, how important Lisp's
uniformity of syntax is, for its macro system. Generally speaking, a
macro accepts as arguments some mixture of data, identifiers and
expressions (let's forget about the existence of Python statements for
now, they complicate the situation even more). How do you pass in a
mixture of data and code?

Data is usually presented in a tuple:

   foo(datum1, datum2, datum3) 

Expressions are usually presented on separate lines:

   def foo(...):
       expression1
       expression2
       expression3

Let's try to think of what a Python with-open-file macro call would
look like.

In CL, with-open-file takes

1) a list containing an identifier (a symbol), a pathname (let's call
   it a string) with some optional keywords.

2) arbitrary code

It then binds the identifer to a stream associated with the file, and
expands the source code inside an exception handling environment.

So, the expansion of a (simplistic) Python with_open_file might look
like this:

    outfile = file("foo/bar", "w")
    try:
        print >> outfile, foo(...)
        for i in range(10):
            print >> outfile, bar(...)
    except:
        perform the necessary closing and cleaning up

But what should the corresponding call to with_open_file look like?

We have to pass in the identifier "outfile" (this is where it is very
handy to have symbols as a first-class data type, as in Lisp), and the
file name "foo/bar" ... but then we also have to pass in the chunk of
code 

    print >> outfile, foo(...)
    for i in range(10):
        print >> outfile, bar(i)
    
In Lisp it would look something like this:

    (with-open-file (outfile "foo/bar" :direction :output)
      (format outfile "~a" (foo ...))
      (dotimes (i 10)
        (format outfile "~&~a" (bar i))))

Now, I refuse to contemplate passing the source code in a string, in
the Python version.

Maybe we have to accept the distinction between data-like data, and
code-like data, having the macro accept both a tuple of arguments, and
blocks of code, so the call might look thus:

    with_open_file("outfile", "foo/bar", "w"):
        code:
            print >> outfile, foo(...)
            for i in range(10):
                print >> outfile, bar(...)

(If you need to pass in more than one code block, each one would go in
a seperate "code:" block.)

What could the definition look like?

    defmacro with_open_file(stream, filename, mode)(block1)
        expand(stream) = file(filename, mode)
        try:
            expand(block1)
        except:
            perform the necessary closing and clearing up

(expand would be a function which gets replaced by its argument, at
macro expansion time.)

Hmm, just about bearable, I guess, but I doubt that it generalizes
very well.

Note that we didn't transform the source code argument, just pasted it
in directly. If we wanted to manipulate it first, we'd be in for
serious suffering.

> You could clearly accomplish this functionality using other means,

I think it is best to steer clear of such examples, as experience
shows that some people will not be able to resist the temptation to
conclude (and suggest) that there is no point to macros, and the
discussion gets derailed. We've seen enough of "you could do this
quite easily without macros, but ...", and not enough of "This would
be soooo painful without macros ..." (I won't say "impossible", as
there is always Turing completeness.)

What was wrong with my original examples? I'd like to know, so that I
can replace them with better ones.