do...until wisdom needed...

[Douglas Alan]

"Andrew Dalke" <dalke at acm.org> writes:

> Douglas Alan replied:
> >Procedural macros are best left to the experienced professional
> >programmer to implement.  The more casual programmer can significantly
> >benefit from the work of the expert, though.

> Hmmm. I am an experienced professional programmer, at least for my
> field.  And I don't want to touch procedural macros.  From what I've
> heard from others in private mail, too often non- "experienced
> professional programmers" end up making their own dialect, and the
> result is a Babel of code.

The same thing is true when non-"experienced programmers programmers"
make a new library module of any sort.  Any bad library can be a
nightmare to maintain and propagate bad coding style.  I've seen quite
the doozies in my time.

> As one person
> commented to me:

> ] If you give people extensible syntax, and Andrew and I adopt
> ] different dialects of python, then instead of getting a bug fix, I
> ] will get, '`I am sorry, but I don't have time to learn your macro
> ] system.' or even worse, `I haven't used regular python for so
> ] long, that I forget how it works'.  This is precisely what
> ] happened to me [..when I..] decided writing my own lisp macros to
> ] do physics in was a good idea.  It was a terrible, terrible,
> ] horrible mistake.  I doomed myself to doing ALL the physics in the
> ] world.

Same thing happens without macros.  I use all sorts of library
routines in my code that I implemented to make my life much easier.
Then when someone comes to me that doesn't have my group's module
library, I either have to have them take our module library, or I have
to program in a hamstrung manner.

> My main expertise is developing infrastructures that allow widely
> different tools to interoperate.  As such, I aim to reduce the
> amount of variety so people can work on getting their research done,
> not figuring out how to use the tools.  Nearly everything I hear
> suggest that procedural macros don't help reduce complexity.

All tools can be used for good or evil.  You use the tool wisely, it
makes your life easier; you use the tool foolishly and you make life
difficult for you.  If it's a good tool, I'd prefer to have the tool
and the choice than be denied the tool.

> I still maintain that having the Python source code available gives
> at least comparable advantages to procedural macros.  If someone
> wants to implement new behaviour, it can be done without all that
> much work.  The big advantage is that all of these changes don't get
> propagated willy-nilly.

Reuse by diverging source code is extremely bad.  It's the worst
maintenance nightmare imaginable.

> Either you take the hit of distributing a new Python-derived
> language (as with Alice) or it's good enough that Guido accepts it
> into the core, in which case it isn't a dialect.  In either case, it
> makes you think strongly about the worth of the new language
> extension.

Most likely the steep slope of that approach makes you throw up your
hands and do things in a less productive and harder to maintain
manner, no matter how much gain the extension would provide.

> Yes, doing this required using a non-Python language for the
> implementation, but as I've said I don't believe in restricting
> myself to one language.  Yes, it reduces the ability of just anyone
> to modify the language, but you are already saying the only ones
> qualified to change a language are "experienced professional
> programmer"s, and they should have no problem with C.  Yes, it calls
> for more work to implement, but I'm not convinced it should be easy.

There's just no way that I would modify the Python source for any
project that I was working on, unless absolutely necessary.  And even
then I'd hand my head in shame and consider myself criminally
negligent.

> Other ways to get around the lack of procedural macros were
> mentioned (besides changing the language core):
>    - calling a library function to parse an alternate language
>    - importing the new code via a modified import hook
>        (this could be made easier with the work on a library
>         to build ASTs for Python)
>    - writing a C extension which uses the metaobject interface,
>         as with the Persistence type in Zope.  (This can only
>         change the object behaviour.)

Well, no doubt what I'd do is just write a preprocessor for Python
that implements procedural macros.  But I'd feel bad about doing this
-- it would be difficult to maintain Python's RAD interactive feel
without spending a lot of effort on it.

> So who all is affected by the lack of procedural macros in a way
> that cannot be supported by one of these mechanisms?  I suspect it's
> a small audience.  Not to say it wouldn't be an influential
> audience, just that it would be small.

It wouldn't be small if Python had procedural macros.  Most people
would love them -- just like most people love them in Lisp.

> > The same thing is often true for classes

> That's true.  My clients don't have much of a problem using
> classes/instances, but the ones without much CS training are
> somewhat hesitant writing their own classes.  But the complexity of
> classes seems to be much less than procedural macros, since people
> who aren't professional programmers can still learn to write them
> well enough.

I think classes are much more complicated conceptually than procedural
macros -- learning how to use data abstraction, inheritance, and
polymorphism properly is no small feat.  Complicated macros can be a
bit tricky to actually code and debug, but there's nothing
conceptually difficult about them.

> Me:

> >> I can also point out the almost complete nonexistence of a major
> >> package in computational chemistry or biology written in a language
> >> supporting hygenic/procedural macros as an indicator that
> >> *something* prevents their wider use in my field.

> >There aren't any widely used languages that have procedural macros
> >other than Lisp and its variants.

> That's almost my point, but the other way around.  If procedural
> macros are the cat's meow then you'ld think that someone else beside
> the Lisp'ers would implement it.

People who haven't programmed in Lisp just don't know what they're
missing, so they don't give it much thought.  Look how long it took OO
to catch on (another invention that grew out of Lisp).  Most of the
arguments people make against procedural macros today, people used to
make about about OO.

> So either a) new ideas take a long time to become mainstream, so
> wait another decade,

Indeed they do, but who wants to wait.

> b) the rest of the world is too ignorant/ conservative/
> unambitious/ etc.

Well, that's why it takes so long.  OO was invented in the '70's and
it took until the '90's for it to catch on.

> c) there is a problem with the usefulness of procedural macros in
> mainstream development

They've never been used in "mainstream development" -- except to the
extent that Lisp (and Dylan) have been used for mainstream
development.  Lisp succeeds well when it is used.  The reason it is
not more widely used as nothing to do with its macro mechanism.  The
main reason, I think, is that many people just don't like Lisp's
syntax, which is a bit peculiar.  The other reason is that it was far
ahead of its time: it used GC and was a memory hog for its time and
was something of a challenge to implement efficiently.  These things
gave it a bad rep back when most people wanted to program in Fortran
or assembly or later C.  It never recovered from this bad rep.  GC was
the death knell for any language in the mainstream, until Java, and
that's very recent.

> or d) it cannot be done in any language other than Lisp.

That's not the case -- it's just not quite as "natural" since Lisp
syntax just screams out for it.  Lisp's syntax just made it
*inevitable* that procedural macros would be invented for it, while
the syntax of most other languages made it inevitable that the thought
would never come to mind (except to someone who had already used
Lisp).

> My perception of things weighs things more towards c).  It seems
> that you weigh b) stronger than I do, with some complaint about a).
> I still can't figure out how address the d) part without basically
> quoting everything, in which case it's identical to having a
> function that parses a string and exec's the result.

Dylan proves that it can be done.  (It's derived from Lisp, but has a
more traditional syntax.)  Macros are typically executed at
compile-time, not run-time, which is often necessary for efficiency
reasons.  Dylan didn't catch on because it was a new language without
a solid niche that desperately needed filling.  (The only way a new
language succeed is if people are *desperate* for something to fill a
neglected niche.)

> > And Lisp only really ever caught on in AI circles and for teaching
> > programming at finer institutions of higher education.  The
> > reasons for that are complex and I think don't say much about the
> > pros and cons of procedural macros, or even of Lisp in general.

> "finer institutions of higher education"?  Seems a rather biased
> phrase.  Bet my alma mater's football team could beat yours! :)
> And its music department, and its film school, and its synchronized
> swimming team, and its hospitality management program, and its
> school of library science ... oh yeah, and its magnet lab (sorry,
> that's an eight year old jibe :)

Oh, I was just using a funny phrase like from a TV commercial.  In any
case, Scheme (a dialect of Lisp) is usually considered state of the
art for teaching Computer Science 101.  Scheme is what's used at MIT,
Harvard, Yale, etc.

|>oug