Python syntax in Lisp and Scheme

[Andrew Dalke]

Peter Seibel:
>So, to write a new test function, here's what I
> write:
>
>   (deftest foo-tests ()
>     (check
>      (= (foo 1 2 3) 42)
>      (= (foo 4 5 6) 99)))

Python bases its unit tests on introspection.  Including the
full scaffolding, the equivalent for Python would be

        import unittest
        import foo_module  # I'm assuming 'foo' is in some other module

        class FooTestCase(unittest.TestCase):
            def testFoo(self):
                self.assertEquals(foo_module.foo(1, 2, 3), 42)
                self.assertEquals(foo_module.foo(4, 5, 6), 99)

        if __name__ == '__main__':
            unittest.main()

Here's what it looks like

>>> class FooTestCase(unittest.TestCase):
...    def testFoo(self):
...       self.assertEquals(foo(1,2,3), 42)
...       self.assertEquals(foo(4,5,6), 99)
...
>>> unittest.main()
F
======================================================================
FAIL: testFoo (__main__.FooTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<interactive input>", line 4, in testFoo
  File "E:\Python23\Lib\unittest.py", line 302, in failUnlessEqual
    raise self.failureException, \
AssertionError: 42 != 99

A different style of test can be done with doctest, which uses Python's
docstrings.  I'll define the function and include an invalid example
in the documentation.

def foo(x, y, z):
    """Returns 42

    >>> foo(1,2,3)
    42
    >>> foo(4,5,6)
    99
    >>>
    """"
    return 42

Here's what I see when I run it.

>>> import doctest
>>> doctest.testmod()
*****************************************************************
Failure in example: foo(5,6,7)
from line #3 of __main__.foo
Expected: 99
Got: 42
*****************************************************************

Doctests are fun. ;)

> Note that this is all about the problem domain, namely testing. Each
> form within the body of the CHECK is evaluated as a separate test
> case.

The unittest example I have makes them all part of the same test case.
To be a different test case it needs a name.  If it has a name, it can
be tested independent of the other tests, eg, if you want to tell the
regression framework to run only one of the tests, as when debugging.
If you have that functionality you'll have to specify the test by number.

> If a given form doesn't evaluate to true then a failure is
> reported like this which tells me which test function the failure
> was in, the literal form of the test case and then the values of any
> non-literal values is the function call (i.e. the arguments to = in
> this case.)

The Python code is more verbose in that regard because ==
isn't a way to write a function.  I assume you also have tests for
things like "should throw exception of type X" and "should not
throw expection" and "floating point within epsilon of expected value"?

>   Test Failure:
>
>     Test Name: (FOO-TESTS)
>     Test Case: (= (FOO 1 2 3) 42)
>     Values:    (FOO 1 2 3): 6

Feel free to compare with the above. The main difference, as you
point out below, is that you get to see the full expression.  Python
keeps track of the source line number, which you can see in the
traceback.  If the text was in a file it would also show the contents
of that line in the traceback, which would provide equivalent output
to what you have.  In this case the input was from a string and it
doesn't keep strings around for use in tracebacks.

(And the 'doctest' output includes the part of the text used to
generate the test; the previous paragraph only applies to unittest.)

I expect a decent IDE would make it easy to get to an
error line given the unittest output.  I really should try one of
the Python IDEs, or even just experiment with python-mode.

I expect the usefulness of showing the full expression to be
smaller when the expression is large, because it could be
an intermediate in the expression which has the problem, and
you don't display those intermediates.

> So what is the equivalent non-macro code? Well the equivalent code
> to the DEFTEST form (i.e. the macro expansion) is not *that* much
> more complex--it just has to do the stuff I mentioned; binding the
> test name variable and registering the test function. But it's
> complex enough that I sure wouldn't want to have to type it over and
> over again each time I write a test:

Python's introspection approach works by looking for classes of a
given type (yes, classes, not instances), then looking for methods
in that class which have a given prefix.  These methods become the
test cases.  I imagine Lisp could work the same way, except that
because other solutions exist (like macros), there's a prefered
reason to choose another style.

The Python code is the same number of lines as your code, except
that it is more verbose.  It does include the ability for tests to have
a setup and teardown stage, which appears to be harder for your
code to handle.

> Note that it's the ability, at macro expansion time, to treat the code
> as data that allows me to generate test failure messages that contain
> the literal code of the test case *and* the value that it evaluated
> to. I could certainly write a HOF version of CHECK that accepts a list
> of test-case-functions:

> But since each test case would be an opaque function object by the
> time CHECK sees it, there'd be no good option for nice reporting from
> the test framework.

You are correct in that Python's way of handling the output doesn't
include the expression which failed.  Intead, it includes the location
(source + line number) in the stack trace and if that source is a file
which still exists it shows that line which failed.

A solution which would get what you want without macros is
the addition of more parse tree information, like the start/end positions
of each expression.  In that way the function could look up the
stack, find the context from which it was called, then get the full
text of the call.  This gets at the code "from the other direction",
that is, from looking at the code after it was parsed rather than
before.

Or as I said, let the IDE help you find the error location and
full context.

> but for me, the test, no pun intended, is, is the thing I have to
> write to define a new test function much more complex than my original
> DEFTEST form?

I'll let you decide if Lisp's introspection abilities provide an alternate
non-macro way to handle building test cases which is just as short.

Knowing roughly no Lisp and doing just pattern matching, here's
a related solution, which doesn't use classes.

(defun utest-foo ()
  (= (foo 1 2 3) 42)
  (= (foo 4 5 6) 99))

 ...
(run-unit-tests)

where run-unit-tests looks at all the defined symbols, finds
those which start with 'utest-', wraps the body of each one
inside a 'check' then runs the
    (eval-when (:compile-toplevel :load-toplevel :execute)
       ..
on the body.

If that works, it appears to make the unit test code slightly
easier because the 'check' macro is no longer needed in each
of the test cases; it's been moved to 'run-unit-tests' and can
therefore work as a standard function.

                    Andrew
                    dalke at dalkescientific.com