do...until wisdom needed...

[Douglas Alan]

I just read an article, "Being Popular", by Paul Graham, a famous Lisp
guru on why Common Lisp effectively killed off Lisp and what should be
done about the situation.  I agree with most of what it says (I've
said a lot of it myself less eloquently in one form or another over
the years).  It should hopefully satisfactorily answer your question
as to why Lisp never made it in the mainstream:

   http://www.paulgraham.com/popular.html

Here are some excerpts:

     To become popular, a programming language has to be the scripting
     language of a popular system. Fortran and Cobol were the
     scripting languages of early IBM mainframes. C was the scripting
     language of Unix, and so, later, was Perl. Tcl is the scripting
     language of Tk. Java and Javascript are intended to be the
     scripting languages of web browsers.

     Lisp is not a massively popular language because it is not the
     scripting language of a massively popular system. What popularity
     it retains dates back to the 1960s and 1970s, when it was the
     scripting language of MIT. A lot of the great programmers of the
     day were associated with MIT at some point. And in the early
     1970s, before C, MIT's dialect of Lisp, called MacLisp, was one
     of the only programming languages a serious hacker would want to
     use.

     ...

     Common Lisp is unpopular partly because it's an orphan. It did
     originally come with a system to hack: the Lisp Machine. But Lisp
     Machines (along with parallel computers) were steamrollered by
     the increasing power of general purpose processors in the
     1980s.  Common Lisp might have remained popular if it had been a
     good scripting language for Unix.  It is, alas, an atrociously bad
     one.

     ...

     A programming language does need a good implementation, of
     course, and this must be free. Companies will pay for software,
     but individual hackers won't, and it's the hackers you need to
     attract.

     A language also needs to have a book about it. The book should be
     thin, well-written, and full of good examples. K&R is the ideal
     here. At the moment I'd almost say that a language has to have a
     book published by O'Reilly. That's becoming the test of mattering
     to hackers.

     ...

     There is one thing more important than brevity to a hacker: being
     able to do what you want. In the history of programming languages
     a surprising amount of effort has gone into preventing
     programmers from doing things considered to be improper. This is
     a dangerously presumptuous plan. How can the language designer
     know what the programmer is going to need to do? I think language
     designers would do better to consider their target user to be a
     genius who will need to do things they never anticipated, rather
     than a bumbler who needs to be protected from himself. The
     bumbler will shoot himself in the foot anyway.  You may save him
     from referring to variables in another package, but you can't
     save him from writing a badly designed program to solve the wrong
     problem, and taking forever to do it.

     ...

     Historically, Lisp has been good at letting hackers have their
     way. The political correctness of Common Lisp is an
     aberration. Early Lisps let you get your hands on everything. A
     good deal of that spirit is, fortunately, preserved in
     macros. What a wonderful thing, to be able to make arbitrary
     transformations on the source code.

     Classic macros are a real hacker's tool--simple, powerful, and
     dangerous. It's so easy to understand what they do: you call a
     function on the macro's arguments, and whatever it returns gets
     inserted in place of the macro call.

     ...

     A really good language should be both clean and dirty: cleanly
     designed, with a small core of well understood and highly
     orthogonal operators, but dirty in the sense that it lets hackers
     have their way with it. C is like this. So were the early
     Lisps. A real hacker's language will always have a slightly
     raffish character.

     ...

     To be attractive to hackers, a language must be good for writing
     the kinds of programs they want to write. And that means, perhaps
     surprisingly, that it has to be good for writing throwaway
     programs.

     ...

     A popular programming language should be interactive, and start
     up fast.

     ...

     I think a lot of the advances that happen in programming
     languages in the next fifty years will have to do with library
     functions. I think future programming languages will have
     libraries that are as carefully designed as the core
     language. Programming language design will not be about whether
     to make your language strongly or weakly typed, or object
     oriented, or functional, or whatever, but about how to design
     great libraries. The kind of language designers who like to think
     about how to design type systems may shudder at this. It's almost
     like writing applications! Too bad. Languages are for
     programmers, and libraries are what programmers need.

     ...
   
     We can get rid of (or make optional) a lot of parentheses by
     making indentation significant. That's how programmers read code
     anyway: when indentation says one thing and delimiters say
     another, we go by the indentation. Treating indentation as
     significant would eliminate this common source of bugs as well as
     making programs shorter.

     ...    
 
     The most important part of design is redesign. Programming
     languages, especially, don't get redesigned enough.

     ...

     Users are a double-edged sword. They can help you improve your
     language, but they can also deter you from improving it. So
     choose your users carefully, and be slow to grow their
     number. Having users is like optimization: the wise course is to
     delay it. Also, as a general rule, you can at any given time get
     away with changing more than you think. Introducing change is
     like pulling off a bandage: the pain is a memory almost as soon
     as you feel it.

     ...

     What all this implies is that there is hope for a new Lisp. There
     is hope for any language that gives hackers what they want,
     including Lisp. I think we may have made a mistake in thinking
     that hackers are turned off by Lisp's strangeness. This
     comforting illusion may have prevented us from seeing the real
     problem with Lisp, or at least Common Lisp, which is that it
     sucks for doing what hackers want to do. A hacker's language
     needs powerful libraries and something to hack. Common Lisp has
     neither. A hacker's language is terse and hackable. Common Lisp
     is not.

     The good news is, it's not Lisp that sucks, but Common Lisp. If
     we can develop a new Lisp that is a real hacker's language, I
     think hackers will use it. They will use whatever language does
     the job. All we have to do is make sure this new Lisp does some
     important job better than other languages.

     History offers some encouragement. Over time, successive new
     programming languages have taken more and more features from
     Lisp. There is no longer much left to copy before the language
     you've made is Lisp. The latest hot language, Python, is a
     watered-down Lisp with infix syntax and no macros. A new Lisp
     would be a natural step in this progression.

     I sometimes think that it would be a good marketing trick to call
     it an improved version of Python. That sounds hipper than
     Lisp. To many people, Lisp is a slow AI language with a lot of
     parentheses.

     ...


     The Dream Language

     By way of summary, let's try describing the hacker's dream
     language. The dream language is beautiful, clean, and terse. It
     has an interactive toplevel that starts up fast. You can write
     programs to solve common problems with very little code. Nearly
     all the code in any program you write is code that's specific to
     your application. Everything else has been done for you.

     The syntax of the language is brief to a fault. You never have to
     type an unnecessary character, or even to use the shift key much.

     Using big abstractions you can write the first version of a
     program very quickly. Later, when you want to optimize, there's a
     really good profiler that tells you where to focus your
     attention. You can make inner loops blindingly fast, even writing
     inline byte code if you need to.

     There are lots of good examples to learn from, and the language
     is intuitive enough that you can learn how to use it from
     examples in a couple minutes. You don't need to look in the
     manual much. The manual is thin, and has few warnings and
     qualifications.

     The language has a small core, and powerful, highly orthogonal
     libraries that are as carefully designed as the core
     language. The libraries all work well together; everything in the
     language fits together like the parts in a fine camera. Nothing
     is deprecated, or retained for compatibility. The source code of
     all the libraries is readily available. It's easy to talk to the
     operating system and to applications written in other languages.

     The language is built in layers. The higher-level abstractions
     are built in a very transparent way out of lower-level
     abstractions, which you can get hold of if you want.

     Nothing is hidden from you that doesn't absolutely have to
     be. The language offers abstractions only as a way of saving you
     work, rather than as a way of telling you what to do. In fact,
     the language encourages you to be an equal participant in its
     design. You can change everything about it, including even its
     syntax, and anything you write has, as much as possible, the same
     status as what comes predefined.

|>oug