Benefits of moving from Python to Common Lisp?

[Marco Antoniotti]

Manoj Plakal <plakal at cs.wisc.edu> writes:

> Marco Antoniotti wrote:
> 
> > "Morten W. Petersen" <morten at thingamy.net> writes:
> > Apart from the existence of very good native compilers, there are a
> > number of nice features - all standard - that allow you to use Common
> > Lisp as a "language kit": something very few other language are good at.
> > 
> > Take for example one of the latest additions to Python: list
> > comprehension.
> > 
> > This is essentially incorporating regular set-theoretic notation in
> > the language. I.e. to have notation that allows you to say things like
> > 
> > 	{x | even(x) /\ x >= 0 /\ x <= 100}
> > 
> > (the set of "numbers" between 0 and 100 - assuming integers or rational,
> > this is a finite, enumerable set).
> > 
> > To get this kind of notation into Python (alas a similar one), AFAIU,
> > the (1) parser had to be modified, and (2) the interpreter had to be
> > modified as well. This are not things that (AFAIK) where done in
> > Python itself.
> > 
> > In Common Lisp you just changed the "readtable" to recognize the
> > special nature of the characters #\{, #\}, #\[, and #\] (think of
> > Emacs and what it does with characters) and you write a couple of
> > macros.  The code is all written in CL and the above example may be
> > rendered as
> > 
> > 	{x / x in (range 0 100) / (evenp x)}
> > 
> > This effectively constructs a list of the even integers.  All of this is
> > all CL and it is all compiled down to native code. (The above expression
> > "expands" into a LOOP, which is happily compiled by the CL system.  The
> > code runs on all CL implementations and it can be used to shorten your
> > programs.
>  
> 
>           I'm curious (I haven't done much programming in Lisp).
> 
>           Doesn't this make CL programs harder to read? I can see
>           why a programmer would love it since it allows you to
>           define your own little language that fits your way
>           of thinking and a problem domain. I've heard of Lisp
>           programmers going gaga over the macro facility.
> 
>           But isn't it going to be difficult for someone other
>           than the original programmer to come along and try to
>           figure out what's going on? They're going to have to absorb
>           the macro definitions before trying to read even
>           a small part of the code that uses these macros.

I believe you make a very good point w.r.t. the "SetL" package for
CL.  That is, the extension to CL I wrote to use set theoretic
notation within CL programs.  When using it, you are writing in a "non
traditional" CL/Lisp notation, hence, your programs may not be
"readable" by people who are not familiar with the notation.

The "SetL" package was written to illustrate a capability of CL
systems and to provide a package for people who want to use it.  If
you want to use it, you'd better advertise that you are using it and
point to the relevant documentation (which I guiltily did not provide :) ).

In a more "traditional" setting, writing macros in CL is a useful
thing, because "regular" macros do not - in my experience - deviate
much from "traditional" CL code.

Let's take the (standard) macro `defclass'.  You write

	(defclass my-class (super-class-1 super-class-2)
           ((slot1 :accessor slot1 :initarg :s1)
            (slot1 :accessor slot1 :initarg :s1)))

this is usually defined as a macro and its syntax is "traditionally"
CL.  A different macro is

	(defsystem bio-l
	   :source-pathname "BIOL:"
           :components ((:file "biol-package")
		        (:module "sequences"
                                 :components ("seq" "alignments"))))

`defsystem' is another macro (in the publicily available "MAKE"
package).

All in all when you see a `def*' form in a CL program, you know that
(most likely) you are looking at a macro.  Hence you have a visual
clue.

Other "traditional" macros are of the form `with-*'.  THe typical
example is `with-open-file'

	(with-open-file (f "the-file")
           (read f))

This effectively expands into something like

        (let ((f (open "the-file"
                       :direction :input :if-does-not-exist :error)))
           (unwind-protect
              (read f)
              (close f))))
		
I.e. the macro is used to ensure that some "cleanup" form is
effectively used.

In my experience, the most common forms of macros fall three
cathegories: defining macros, with-* style macros and "efficiency"
macros.

Of course, you have other forms of macros.  Large packages give you
several macros that you must understand before you use them.  But in
these cases (I am thinking, e.g. of the SERIES package), you know that
you need to learn how to use them.

>           A similar thing happens when
>           people go nuts with the C preprocessor or C++
>           operator overloading. You get either innocent-looking
>           code that actually has very different semantics
>           than what you thought, or completely baroque code
>           which can only be understood by reading a whole
>           lot of other code that is defining macros and stuff.
> 
>           How much of a problem is this in practice for CL?

As I said before, macro use in CL is extremely powerful, but it is
also used in some "patterns".  This helps.

Operator overloading in C++ has been the source of misunderstandings.
I think this is because of the initial confluence of "namespace" and
"classes".  In CL classes are not used for namespace purposes.  CL
packages are like (well, almost) Ada packages and newer C++
namespaces.  Moreover, to "overload" a function you must either design
it to be like that from scratch or go against the standard.  Let's
take an example.  Writing a "Matlab-like" math package, requires you
to redefine '*' to take matrix operands.

>From the point of view of the programmer this requires to state at the
package level that you are "shadowing" the standard '*' operator.

==============================================================================
(defpackage "MATRIX" (:use "COMMON-LISP")
  (:shadow cl:*) ; Note the explicit reference to the standard '+' operator
  (:export "*")
  )
==============================================================================

Then you need to write *in the MATRIX package* the "overloaded" version.

==============================================================================
(in-package "MATRIX")

(defmethod * ((n1 number) (n2 number)) ; This '*' is really matrix:*
  (cl:* n1 n2))  ; Explicit reference to the standard '*'.

(defmethod * ((n number) (m matrix))
   ;; code for scalar product
   )

(defmethod * ((n complex) (m matrix))
   ;; code for scalar product, specialized on complex scalar.
   )

(defmethod * ((m matrix) (n number))
   ;; code for scalar product; commutative version
   )

(defmethod * ((m1 matrix) (m2 matrix))
   ;; code for matrix product
   )

...
==============================================================================

Now, this allows you to "overload" the '*' operator.

>From the "user" point of view, he needs to know what s/he can use.  If
s/he decides to work in the MATRIX package, s/he will have to know
that '*' has been "extended".  IMHO, in cases like this, CL offers
more control over what you see.  Add to this the tendency of CL
programmers to use very-long-and-explicative-names-for-their-functions
and you see how the problems are tamer that they may seem.

Of course your mileage may vary, and "RTFM" is the commandament you
must follow :)

Cheers


-- 
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group        tel. +1 - 212 - 998 3488
719 Broadway 12th Floor                 fax  +1 - 212 - 995 4122
New York, NY 10003, USA                 http://bioinformatics.cat.nyu.edu
                    "Hello New York! We'll do what we can!"
                           Bill Murray in `Ghostbusters'.