Explanation of macros; Haskell macros

[Peter Seibel]

Joachim Durchholz <joachim.durchholz at web.de> writes:

> Peter Seibel wrote:
> 
> > Joachim Durchholz <joachim.durchholz at web.de> writes:
> >
> >>Peter Seibel wrote:
> >>
> >>>Joachim Durchholz <joachim.durchholz at web.de> writes:
> >>>
> >>>>And, of course, macros can evaluate at compile time. Personally, I'd
> >>>>prefer to do compile-time evaluation based on "the compiler will
> >>>>evaluate all known-to-be-constant expressions". The advantage here
> >>>>is that programmers don't need to learn another sublanguage for
> >>>>compile-time expressions.
> >>>
> >>>Ah, but in Lisp we don't have to. We use Lisp.
> >>
> >>Having readers and special forms /is/ an extra sublanguage. I don't
> >>have to learn extra syntax for these forms (which is good), but I do
> >>have to learn about a lot of special rules that apply to macros and
> >>nothing else (which is not so good).
> > Hmmmm. The special forms (25 of them, called special operators these
> > days, by the by) are used the same in macros and functions. Lisp's
> > customizable reader is a separate thing--there is no need to customize
> > the reader to write macros.
> 
> You're right, and made me rethink what's actually disturbing me about
> macros.
> 
> Perhaps it's that I have to adapt to dual-mode (two-tier?) thinking:
> I have to reason about both what the macros are doing and what the
> software is doing. Alternatively, I could consider the macros as
> "part of the language" and not reason about the macro code but about
> their effects - in which case I have effectively augmented the
> language by all macros that are in use.

I think you're right that you have to adopt a dual-mode of thinking.
When you're writing macros you're essentially extending the
compiler/language to recognize constructs that would otherwise be
meaningless.

Then, having written them, you use them as if they were in the
language all along. At one level this is no different from extending a
language by writing a function. That is, if you're working in a
language (like Common Lisp, Scheme, C, or--I imagine--any FPL) where
much of the "language" itself is implemented in terms of built-in
functions, if you write a new function you're extending the language
and after you've written it you can forget about the details of how it
works and just use it. The only difference between macros and
functions is that macros operate by generating code which then
performs actions as opposed to *being* code that performs actions. But
that level of indirection makes it easy to express things that would
otherwise be difficult (in my experience.)

> Personally, I'd still prefer a compiler that's evaluating constant
> expression.

Hmmm. If it will make you feel any better, macros are just fuctions
whose domain and range happens to be Lisp expressions. That happen to
be run by the compiler. So eventually the compiler is evaluating
constant expressions, just some of them were automatically derived
from the written source.

> Is there anything that a macro does that can't be done by
> preevaluating data structures that contain functions (or closures)?
> At first glance, I'd say no, but then I don't know what macros are
> used for in practice.

Well it depends whether you consider syntax to be "anything". I think
it was you who objected to one of my examples by saying, "that's just
syntactic sugar". Macros can (and many do) do large amount of
under-the-covers bookkeeping. For instance here are a few rules from a
grammar for a lexer for Java source code:

  (defprod line-terminator () (/ #\newline (#\return (? #\newline))))

  (defprod white-space () (/ #\space #\tab #\page line-terminator))

  (defprod input () ((* input-element) (? #\Sub)))

  (defprod input-element () (/ white-space comment token))

  (defprod token () (/ identifier java-keyword literal separator operator))

DEFPROD is a macro that squirrels away the stuff on the right which is
an s-expy form of BNF. The rest of the grammar is more of the same. At
the bottom of the grammar file where the productions are diffined I
have this form:

  (deflexer java-lexer (input)
    ((:tokens identifier java-keyword literal separator operator)))

That DEFLEXER call (another macro) expands into a single parsing
function built out of all the productions created by DEFPROD calls,
appropriately wired together and embedded into code that takes care of
the stepping through the input and gather up values, etc. And that
function is compiled into extremely efficient code because all the
intercommunication between productions goes through lexical variables.
And the file containing these calls to DEFPROD and DEFLEXER is legal
Lisp source which I can feed to the compiler and get native machine
code back.

So I don't know if that is "anything" or not. I don't know how I would
write such a thing in Haskell, et al. but I know this is a *lot*
cleaner than what *I'd* be able to do in Java, Perl, Python, or C.

> Um, well, yes, there is one thing that macros can do: extending
> syntax in ways that aren't part of the original language syntax.
> E.g. replacing all those parentheses by indentation, or something
> similar un-Lispish. (Extending syntax in such ways is a mistake
> IMHO, but YMMV. Anyway, I'm more interested in the question if
> there's any /semantics/ that can be done via macros but not via
> compile-time evaluation.)

Actually, changing the syntax is--if one thinks one must--is really
done by read-macros which are quite different. But most Lispers agree
with you--there's just not enough benefit to changing the syntax to be
worth it. Except for occasionally making a new syntax for expressing
certain frequently created literal objects that otherwise would
require a much more verbose creation form. (Someone gave a great
example the other day in another thread of an airline reservation
system (Orbitz I think) that has a special kind of object used to
represent the three-letter airport codes. Since they wanted to always
have the same object representing a given airport they needed to
intern the objets with the TLA as the key. But rather than writing
(intern-airport-code "BOS") everywhere, they wrote a reader macro that
let them write: #!BOS. Since this was an incredibly common operation
in their system, it was worth a tiny bit of new syntax. But note,
again, that's not *changing* the syntax so much as extending it.)

> >> Letting the compiler evaluate what it can means that I don't even
> >> have to learn extra forms.
> >
> > I'm not sure what "extra" forms you're talking about. Other than
> > DEFMACRO, I guess. But by that argument we'd be better off without
> > DEFUN too because that's just another darn thing to learn.
> 
> I believe it's not DEFMACRO that's complicating things, it's the
> macros that it allows (see above).

Fair enough. But do you object to the ability to write new functions
on the grounds that that just means you have a lot of new functions to
learn and that complicates things needlessly? That's obviously a
rhetorical question but I am actually curious why you find them
different, if you do.

> >>I just suspect that better solutions are available in every case,
> > Interesting. A lot of people suspect that who haven't actually used
> > Common Lisp macros. Yet almost all Common Lispers--who by in large are
> > *not* monolinguists--think macros are one of Common Lisp's great
> > features. I'm not saying your wrong, but if those better solutions are
> > out there for all the things I can do with macros, I haven't seen
> > them. Now I don't know Haskell or ML so I'm also suffering from finite
> > knowledge. Maybe one day I'll have time to learn one of them for
> > myself and see if they really do offer better solutions.
> 
> Agreed on all accounts (except that I don't know how "multilingual"
> Lispers really are *g*).

Well, there is the problem that once folks find Lisp they tend to stop
looking for better things because what could be better than Lisp. ;-)
But most Lispers take a fairly circuitous path to Lisp and hit a bunch
of other languages before they find it.

> Does anybody have a keyword-style list of useful applications of the
> macro facilities?
> 
> >>and I not just suspect but know that macros have some very serious
> >>disadvantages (such as bad debugger interaction, a potential for
> >>really ugly hairballs, and a constant temptation for stopgap
> >>solutions that "work well enough").
> > Well, of those the debugger interaction is perhaps the most serious.
> > Yet in practice (Hey Pascal, I almost said "in 99% of cases"!)
> 
> "99% of all cases" is a pretty good argument actually :-)
> It's just that Pascal doesn't (want to) believe that it's enough for
> type checking. His problem, not mine...
> 
>  > it
> > doesn't seem to be that much of a problem. Maybe that's because we've
> > just learned to deal with the pain; maybe MACROEXPAND is all you
> > really need to get your bearings. At any rate, there's no in principle
> > that a Lisp implementation couldn't keep track of macro information
> > along with the compiled code just the way most compiler keep track of
> > line number information in order to show you the code as written in
> > the debugger. (And if it was really slick to let you step through the
> > macro expansion, etc.)
> 
> Agreed. I'm getting more and more convinced that it's not language
> size or KISS issues that's setting me off, it's that "two-tier
> thinking" that I (perhaps mistakenly?) associate with macros.

The funny thing is to me, when you say "two-tier thinking" that
perfectly describes how I think about the process of making
abstractions. Regardless of the *kind* of abstraction one is creating,
one has to be facile at switching mental gears between *building* the
abstraction and *using* it. You are probably so used to doing this
when writing functions that you don't even notice the switch. But
because macros are a bit strange you *notice* the switching and it
annoys you. I suspect that anyone who's capable of building functional
abstractions would--if they actually used macros--quickly learn to
switch gears equally smoothly when writing and using macros.

> >>Lisp-the-language is a quite pretty lean-and-mean KISS language. The
> >>stuff that's built on top of it (macros, readers, dispatch
> >>mechanisms, etc. etc.) is neither lean nor KISS nor (IMHO) pretty -
> >>YMMV.
> > Clearly. I find Common Lisp to be a pretty beautiful piece of
> > *engineering*.  Which may be different than a beautiful realization of
> > a beautiful theory.
> 
> I wouldn't want HM typing if it were just beautiful theory.
> HM typing happens to be a beautiful theory. Things are getting less
> beautiful once you interact with the Real World (TM), which is
> stateful - OTOH, Real World is a mess, so don't expect computing to be
> beautiful anymore when there is interaction with it *g*. What
> surprised me is how much of a computation can be separated from such
> interaction. With the proper framework, one can even describe
> interaction patterns (which are themselves stateless), feed these
> patterns to the framework, and watch in amazement how the execution
> engine follows these patterns. It's the kind of abstractive facility
> I've been yearning for decades...
> Lisp could do this just as well. It's just not done because taking the
> shortcut and doing stateful computations directly is so much easier.
> (And I don't pretend that functional languages are doing this kind of
> thing perfectly right now. I think the potential in these ideas is
> just beginning to be exploited - and what's available is already quite
> impressive.)
> 
> >>Or, more to the point: I have yet to see something that cannot be
> >>done in a leaner, more KISS way.
> > Well, if I promise to continue to think that someday I really should
> > learn a hard-core FP language so I can see what all the static typing
> > fuss is about, will you promise to think in the back of your mind that
> > maybe someday you should learn Common Lisp and see what makes us all
> > so gaga over macros.
> 
> Actually I'm trying to understand macros and macro usage right now,
> without having to learn all the details of CL (which would be a bit of
> overkill - I know it might not be enough, but then my time is limited
> so I'm doing my best within the available budget).

Sure. Cheers.

-Peter

-- 
Peter Seibel                                      peter at javamonkey.com

         Lisp is the red pill. -- John Fraser, comp.lang.lisp