merits of Lisp vs Python

[Steven D'Aprano]

On Sat, 09 Dec 2006 21:55:19 +0000, Kirk Sluder wrote:

> The question I have is why do critics 
> single out macros and not other forms of abstraction such as 
> objects, packages, libraries, and functions?

Who says they do? All forms of abstraction have been criticized. Sometimes
the criticism is valid. Sometimes it warns against abuses of the
abstraction. Just because a feature is useful sometimes doesn't
necessarily mean the benefit outweighs the cost.


> just as an example:
> from foolib import *
> bar.bar("somefile")
> 
> What does this program do? I have no idea. Its functionality is 
> hidden behind multiple layers of abstraction (function, object, 
> library.)

Sure.

But at least you know that foolib is a module or package. You know what
from and import do, and that can't change. And you know that bar is an
object with a method also called bar, it is being called, and the
argument is a string "somefile". Those things can't change. Imagine a
hypothetical language where those two lines could mean *anything*.

Rightly or wrongly, people fear that Lisp's macros push Lisp closer to
that hypothetical anything-goes language than is healthy. Maybe that's a
problem of perception rather than a problem of fact, but you have to ask,
why do people perceive Lisp that way?

Could it be because of people like J Shrager who writes things like this?

"Can't you just expand the language via macros to create whatever facility
of this sort [major new features with new syntax] you need..."

(This thread, dated 8 Dec 2006 16:14:44 -0800)

Or Wolfram Fenske:

"All the interesting features that haven't originated from Lisp (e. g. OO
from Smalltalk) could in turn easily be implemented in Lisp with a couple
of macros."

(This thread, dated 8 Dec 2006 23:38:02 -0800)

To someone outside of Lisp, that looks like "I can make Lisp look like any
language I like in just a few lines." And that implies that to read a Lisp
program, one might need to be able to read code that looks like Lisp, or
Smalltalk, or Prolog, or Fortran, or all of those, or whatever bizarre
syntax the developer wanted it to look like.

My point isn't whether or not their claims are correct (a "couple" of
macros? really?) but that things like this feed the perception that Lisp
is close to that hypothetical language where anything could be anything.
If anything could be anything, do you really know what (+ 1 2) means
without reading every line of code?

And it isn't really that much comfort to be told that good Lisp
developers know not to do stupid things with macros. Sure. But in the real
world of programming, most developers aren't good developers, they are
merely average -- in fact fifty percent of them are below average. And
the bottom 30% can't be trusted with facilities like Lisp's macros,
because they WILL do stupid things with them and turn your code base into
a morass of shifting syntax, buggy DSLs and other maintenance nightmares.
The damage they can do with operator overloading is minor by comparison.

Even something simple like file I/O can be abused. Example: I've seen
*real* code where somebody needed a dict with ten million keys. (Well, he
thought he needed that dict, I never worked out exactly why. Maybe he
thought he was pre-allocating memory as an optimization.) So he started
off with something like this:

def make_dict():
    mydict = {}
    mydict[0] = 0
    mydict[1] = 0
    mydict[2] = 0

and realised that it would take him forever to write that much code. So
he wrote a C program to create the make_dict function for him! Translated
into Python:

def make_dict_factory():
    code = open('makedict.py', 'w')
    code.write("def make_dict():\n")
    code.write("    mydict = {}\n")
    for i in range(10000000):
        code.write("    mydict[%d] = 0\n" % i)
    code.write("    return mydict\n")
    code.close()
    from makedict import make_dict
    return make_dict

(This is an interesting demonstration that any language that allows file
I/O and importing of external program files can always treat functions
as data, even if the language doesn't directly support it. An alternative
would be to keep the strings in memory instead of writing to a module,
then use exec on them instead of importing the module.)

Honest to god, the code really was like that! Maybe I've got the precise
details wrong, it was a long time ago, but if it wasn't just as I've got
it, the consequences were the same: he built a single function that was so
large that loading it caused his machine to thrash for twenty minutes
trying to free enough memory.

Is that an argument against factory functions? Damn straight it is:
they are a powerful tool, and in the hands of morons, they can be
dangerous. Does that mean that languages shouldn't permit higher-order
functions? Not necessarily: all programming tools can be misused, but some
can be misused more easily than others. Power and risk is often a
trade-off, and language designers can't eliminate all risk of stupid
behaviour, but they can design the language to allow whatever level of
risk they believe is acceptable. (E.g. there is no doubt that C's raw
pointers are powerful, but many languages deliberately don't use them.)

The risk of stupid factory functions is small compared to the benefit, but
maybe there is some domain somewhere where the ideal solution is a
language that DOESN'T treat functions as first class objects, deliberately
weakening the language so that a particular class of errors (or stupid
behaviour) just cannot happen. But that language isn't Python.

When it comes to Lisp's macros, the perception is that the power is
correspondingly greater, and the risk of abuse even more so. The safe
examples of what macros can be used for don't seem to provide any
over-whelming advantage (that's not to say they provide NO advantage,
merely that if you use macros safely, the advantage seems small, but if
you use them to their full potential, the risk of abuse is too high).
That's the perspective of many people, and maybe it is wrong. Maybe you
really need to be immersed in Lisp for a while to see the advantages of
macros.

Or maybe it is only an advantage while Lisp programmers are a
self-selected group of above-average skill. Wait until fifty million VB
code monkeys start writing Lisp macros and maybe, just maybe, you'll wish
they were using a less powerful and more restrictive language.



-- 
Steven.