What's better about Ruby than Python?

[Jacek Generowicz]

You wrote lots. Forgive me if I don't address everything. I wish I had
the time to address all your points more carefully.

"Andrew Dalke" <adalke at mindspring.com> writes:

> However, at present, the only example I've seen for when to use a
> macro came from a method cache implementation that I could implement
> in Python using the normal class behaviour,

Or using directors, or lexical closures. Similarly in Lisp, there is
more than one way to do it, and many would not choose the macro option
when implementing a memoizer.

> so I don't have a good idea of when macros would be appropriate
> *for* *Python*.

(Looks like you've found a need for them yourself - see later on.)

Well, frankly, in the context of Python, I find the whole discussion a
bit abstract. Lisp macros fundamentally rely on the fact that Lisp
programs are represented as lists, and that Lisp includes excellent
support for manipulating such lists. The consequence of this (source
code being a form of data) is that it is extremely easy to manipulate
Lisp source code within Lisp. You can already achieve similar things
in Python by typing your source code as a string, and performing
string manipulations on it, and then calling eval ... but it is many
orders of magnitude more painful to do it this way.

Alternatively, get your hands on the parse-tree (I believe there's a
module to help), and mess around with that. That should be much easier
that playing with strings, but still much more of a pain than Lisp
macros.

> When this topic has come up before, others mentioned how
> macros would theoretically be able to, say, modify list.sort
> to return the sorted list after it has been modified in-place.

You don't need macros for that. With the advent of new-style classes,
you subclass list, override the sort function, and replace
__builtins__.list (maybe there's more to it, but as this is not
something I ever intend to do, forgive me for not checking the
details.)

> Given the not infrequent request for the feature, I know that
> if it was allowed, then some of my clients would have done
> that, making it harder for me to know if what I'm looking at
> is core Python behaviour or modified.

That's not the point of macros. The point is not to modify existing
behaviour behind one's back. The point is to add new behaviour
... much like it is with functions, classes etc.

> What you say is true, but most of the code I look at is
> based on fundamental Python types, from which I can
> be assured of fixed behaviour, or classes and functions,
> where I can be assured that they are free to do their
> own thing.  The patterns of behaviour are fixed and
> the opportunities for change well defined.
> 
> Macros, as I understand it, blurs those lines.

I don't think this has anything to do with macros. This is a
consequence of a language allowing to re-bind built-ins. I remind you
that Python allows this, today.

>   I will grant that Lisp or Scheme is the end-all and be-all of
>   languages.

:-) Well, at least that's clear :-) :-)

[NB, I love Python, and wouldn't want to go without it, for a plethora
of reasons.]

> Where's the language between those and Python?

I am not sure that there is a need for one; Python and Lisp are
already very close, as compared to other languages. But maybe some
interemediate language would serve a good purpose.

> Is it possible to have a language which is more flexible than
> Python but which doesn't encourage the various dialectization
> historically evident in the Lisp/Scheme community?

Hmmm. I think this "dialectization of the Lisp/Scheme community" is a
bit like the "dialectization of the Algol Family community" or the
"dialectization of the functonal community" or the "dialectization of
the scripting (whatever that means) community"

The fundamental feature of Lisps is that they represent their source
code in a format which they themselves can manipulate easily. (Often
people try to characterize Lisps by the 4 or 5 fundamental operators
that you need to make all the rest, but I don't find this an
interesting perspective.) What's wrong with there being different
languages with this characteristic? What's wrong with there being
different functional languages? What's wrong with there being
different scripiting (whatever that means) languages ?

Maybe you refer to the fact that, should you wish to make a completely
new Lisp-like language, then starting with an already existing lisp,
and writing your first implementation in that (with the help of
macros), is usually by far the best way of going about it.

(This is exactly how Scheme was created, IIRC)

But your new language is exactly that. A new language. Faithful users
of the language in which you wrote that first implementation, will not
suddenly find that the language they know and love has been broken.

> Could most macros also be written without macros, using
> classes or lambdas?

Heh. You can write a lot of macros which don't need to be macros (so
don't). But there are some things for which macros are absolutely
necessary.

(Actually, in Lisp you could get by with functions and "'" (the quote)
... but you'd still be writing macros, without official language
support for them.)

I guess that it boils down to delaying evaluation until you have had
the opportunity to manipulate your source.

>  How often are the benefits of macros
> that much greater than classes&functions to merit their
> inclusion.
> 
> Does it take skill to know when to use one over the other?

Often, it does. Some are no-brainers (Control structures, for example).

> Do people use macros too often?

Some probably do. This is probably true of classes too.

> When do they hinder misunderstanding?

When they are badly designed. This is true of classes too.

> Are they more prone to misuse than classes&functions?

I guess it is generally true, that the more powerful and versatile the
tool, the more prone it is to misuse.

> The complaint about macros has been their tendency to increase a
> single person's abilities at the cost of overall loss in group
> understanding.

No, no, NO, Noooooooooo !  :-)

At the top of your reply, you agreed about my point that abstracting a
frequently repeated pattern is preferable re-implementing it all over
your code. _This_ is what macros are about.

One can write useless and obfuscating functions and classes, just like
one can write useless and obfuscating macros.

> I've heard references to projects where that didn't occur, but am
> not swayed by it because those seem staffed by people with
> extraordinarily good programming skills almost never found amoung
> the chemists and biologists I work with.

I do not advocate Lisp as a language for people who are not prepared
to invest serious time to understanding it, which probably includes
most people whose primary activity is not programming.

That is why I think Python is _extremely_ useful and necessary. It
provides a significant portion of the power of Lisp, for a small
initial investment.

I do not expect CERN physicists to use Lisp, I do expect them to use
Python.

> The point I'm trying to make is that different, very smart people
> like "Lisp", but insist on variations.

Very smart and not-so-smart people like "scripting languages", but
insist on variations. There's Perl, Python, Ruby ...

> There is clisp

Clisp is just one implementation of an ANSI standardized Lisp (Common Lisp).

> and elisp

Elisp is the scripting language of Emacs.

Now, Common Lisp is an all-purpose stand-alone language designed for
constructing complicated systems; Elisp is designed for configuning
and extending Emacs.

Complaining that this is "insisting on variations of lisp", and that
it is somehow a BAD THING, is a bit like complaining about the
co-existence of Python and Occam, as "insisting on variations of
languages with significant indentation".

> As I understand it, macros can be used to make one lisp variation
> act like another.

This is true to some extent. But just because it can be done, doesn't
mean that very many people actually ever want to do it. Yes, sometimes
CLers want to fake up a Scheme-like continuation, and the language
allows them to do it. Great. But that (pretending to be another
already existing language) is not the point of macros.

> > A few days ago I tested out a C++ library. It didn't work on the C++
> > system I had handy because the STL implementation was
> > different/template support was different. etc. etc.
> 
> Did you really or are your making that up for the
> sake of rhetoric?

Sorry, I should have made clear that I made it up for the sake of
rhetoric. However, the only thing that is untrue is the "A few days
ago" bit. It has happened repeatedly in the past.

> If it takes more than four decades for different Lisp
> implementations to agree on how to import a module, then I think
> there's a problem.

Again, you are confusing "different implementations" with "different
languages".

Implementations of ANSI Common Lisp agree on, well, anything defined
within the standard.

Implementations of different languages clearly do not agree. This is
true of those in the Lisp family, just as it is for members of any
other family of Languages.

> I needed to evalute a user-defined expression where the variable
> names are computed based on calling an associated function.  The
> functions may take a long time to compute and most names are not
> used in an expression, so I want to compute the names only when
> used.

You want lazy evaluation ?

> Did I start from scratch?  No!  I used Python to build the parse
> tree

In Lisp, you start off with the parse tree. That's the great thing
about it.

> then tweaked a few nodes of that tree to change the name lookup into
> the right form, then generated the function from that parse tree.

You've just Greenspunned Lisp macros.

Manipulating the parse-tree is exactly what Lisp macros are about.

> The syntax was the same as Python's, but the behaviour different.

In Lisp you would typically give a name to this behaviour, and then
you would be able to use it alongside the original language.

For example, if the user-defined expression is

  (+ (foo 2) (bar a b c d))

The tweaked version would be

  (lazy-eval (+ (foo 2) (bar a b c d)))

Writing a macro to make Lisp a uniformly lazy language (as you seem to
have been suggesting one might proceed, way up-post), is definitely
not the way to do it.

> Though back at the Python level, it's a "call this function to get
> the needed result",

In Lisp it's "call this macro to get the needed result".

> and has precisely the same nature as any other Python function would
> have.

And has percisely the same nature as any other Lisp macro would have.

> And some languages are not-at-all close to Python.  Eg, I wanted
> to implement a domain-specific language called MCL

What, "Macintosh Common Lisp" ? :-)

> using my PyDaylight package.  I ended up writing a parser for MCL
> and converting the result into Python code, then exec'ing the Python
> code.

In CL this is done with reader macros; a means of altering the way the
parse tree is constructed from the source data.

> Similarly, SMILES is a language for describing molecules.

> Why in the world would I want to extend Lisp/Perl/Python/
> whatever to support that language directly?

a) Because it's easier to build it on top of Lisp than from scratch.

b) Because you never know what future requirements you might have.

> > It's an extreme form of Greenspunning.
> 
> ???  Apparantly not Alan Greenspan.

Greenspun's Tenth Rule of Programming: 

  "Any sufficiently complicated C or Fortran program contains an
  ad-hoc, informally-specified bug-ridden slow implementation of half
  of Common Lisp."

(Of course, it's a general statement about developing in low-level
languages as compared to developing in higher-level ones.)

> Please tell me how you would implement this language
> 
>   load "abc.pdb" into a
>   select "resname LYS" from a into b
>   save b as "lysine.pdb"
> 
> as a macro in Lisp.  I'll assume 'load-pdb' loads a PDB file into
> a object which holds a set of atoms, and that object has the
> method 'select-atoms' which creates a new (sub)set and also
> has the method 'save-as' for saving those atoms in the right format.
> 
> And the above is all that the user can type.
> 
> How in the world does macros make handling that language
> any easier than the standand parser-based non-macro solution?

I don't think I understand the true meaning of your question.


Anyway, your parse tree exapmle shows that you DO understand and use macros
... you just don't know that that's the name of what you are doing :-)