Python syntax in Lisp and Scheme

[Andrew Dalke]

(I know, I swore off cross-posting on this topic, but the claims
made here were too outrageous for me to ignore)

<prunesquallor at comcast.net>
> I wish to show that local information is insufficient for cutting and
> pasting under some circumstances.

Absolutely correct, but rarely the source of errors.  When it does
occur it is almost always immediately after the cut&paste and so
the context is fresh in the mind of the who did the c&p.  The chance
for it to appear in wild code is minute.  I can't recall ever coming
across it.

> Consider this thought experiment:  pick a character (like parenthesis
> for example) go to a random line in a lisp file and insert four of them.
> Is the result syntactically correct?  No.

You're surely exaggerating here.  Here's your factorial example,
which has not been cleaned up.

(defun factorial (x)
(if (> x 0)
x
(*
(factorial (- x 1))
x
)))

 I'll insert four "a" characters in the function name

(defun aaaafactorial (x)
(if (> x 0)
x
(*
(factorial (- x 1))
x
)))

Is that not syntactically correct?  For that matter, what if
I insert four spaces, like

(defun factorial (x)
(if (> x 0)
x
(*
(fact    orial (- x 1))
x
)))

or insert four quotes

(defun factorial (x)
(if (> x 0)
x
''''(*
(factorial (- x 1))
x
)))


> However, there is a class of *syntactic* error that is possible in
> python, but is not possible in lisp

Here's a class of error possible in Lisp and extremely hard to get
in Python -- omitting a space between an operator and an operand
causes an error

Valid Lisp: (- x 1)
Invalid Lisp: (-x 1)     ; ;well, not invalid but semantically different

Valid Python: x - 1
Valid Python: x-1

Yes, I know, it's because '-x' isn't an operator.  Tell it to the
beginning programmer who would have these problems in Python.

Here's another class of error you can get in Lisp but is hard to
get in Python (except in a string).  Randomly insert a '

Valid Lisp: (- x 1)
Valid Lisp: '(- x 1)   ;;; but semantically very different

Will the same beginning user you talk about for Python be
able to identify a single randomly inserted quote in Lisp?

> Now go to a random line in a python file and insert four spaces.  Is
> the result syntactically correct?  Likely.  Could a naive user find
> them?  Unlikely.  Could you write a program to find them?  No.

I tried writing code to test this automatically but ran into problems
because I inserting four spaces at the start of a line may be syntactically
correct and may also not affect the semantics.  For example

def spam(x = some_value,
         y = some_other_value):

Adding 4 characters to the start of the 2nd line doesn't change the
meaning of the line.

There definitely were silent errors, like changing returns of
the sort

    if x > 0:
        return "positive"
    return "nonpositive"

into

    if x > 0:
        return "positive"
        return "nonpositive"

NB: This should be caught in any sort of unit test.  The real test
is if it's hard to detect by a programmer; I'm not the one to
answer that test since I'm too used to Python.  I suspect it is
generally easy to find, especially when the code is actually run,
but that it isn't always automatic.

I also figured given the quote counter example it wasn't
worthwhile categoring everything by hand.

> Delete four adjacent spaces in a Python file.  Will it still compile?
> Likely.

Here's test code.  Note that I give each file 30 chances to succeed
after deleting 4 spaces, and it *never* did so.  That surprised me as
I thought some of the hits would be in continuation blocks.  There
may be a bug in my test code so I submit it here for review.

=================
import string, random, os

def remove_random_4_spaces(s):
    x = s.split("    ")
    if len(x) <= 1:
        # needed for re.py which doesn't have 4 spaces
        return "]]"  # force a syntax error
    i = random.randrange(1, len(x))
    x[i-1:i+1] = [x[i-1] + x[i]]
    return "    ".join(s)

def check_for_errors(filename):
    s = open(filename).read()
    for i in range(30):
        t = remove_random_4_spaces(s)
        try:
            exec t in {}
            print "Success with", filename, "on try", i
            return 0
        except SyntaxError:
            pass
    return 1

def main():
    dirname = os.path.dirname(string.__file__)
    filenames = [name for name in os.listdir(dirname)
                      if name.endswith(".py")]
    count = 0
    errcount = 0
    problems = []
    for filename in filenames:
        print filename,
        err = check_for_errors(os.path.join(dirname, filename))
        if err:
            errcount += 1
            print "properly failed"
        else:
            print "still passed!"
            problems.append(filename)
        count += 1
    print errcount, "correctly failed of", count
    print "had problems with", problems

if __name__ == "__main__":
    main()


=================
anydbm.py properly failed
asynchat.py properly failed
asyncore.py properly failed
atexit.py properly failed
audiodev.py properly failed
base64.py properly failed
  ...
__future__.py properly failed
__phello__.foo.py properly failed
185 correctly failed of 185
had problems with []


                    Andrew
                    dalke at dalkescientific.com