[Distutils] The 'test' command

[Moore, Paul]

From: Andrew Kuchling [mailto:akuchlin@mems-exchange.org]
> An open question: what should the 'test' command do?  We could adopt
> some convention to automatically locate test scripts automatically
> (all files matching test/*.py, for example), with a
> tests=['dir/test1.py', dir/test2.py'] override to explicitly list test
> scripts and ignore the convention.

Yes. That seems correct to me.

> Next, what would it do with the test scripts?  In 2.1b2,
> test.test_support.run_unittest() raises TestFailed when a test suite
> fails.  Would it be sufficient to execfile() all the test scripts,
> note which ones raise TestFailed, and print a list of failing tests
> (which would come after the output of individual test scripts).

I would keep it as minimal as possible for now, so that we can make things
more elaborate in the future without breaking anything.

Suggestion for a specification:
"Each test file will be executed. At the end of the test runs, a summary of
the results will be displayed."

In particular, I don't think we are in a position at this stage to mandate
(or even presume) in what form test scripts should return their results. As
things mature (and in particular, as unittest gets more exposure and use),
standardising on an output format will be easier, and we should be prepared
to take advantage of that. But let's not jump the gun.

As an implementation, what you suggested is pretty good: execfile each test
script, and note any exceptions (I don't see any particular benefit in just
trapping TestFailed here). At the end of the run, print out a summary: xxx
test scripts run, yyy failed (TestFailed exception), zzz terminated
abnormally (any other exception).

This leaves the way open to "improve" things, for example by grabbing the
test output and postprocessing/formatting it, or by changing the format of
the summary. For example, Perl's standard test harness runs test scripts
which print a series of "ok N" messages, but grabs the output and reformats
it as "testname..........m/n" where n is the expected number of tests, and m
gradually increases as the tests run. This is very nice, both as a progress
indicator, and for a "feelgood factor" ("This module ran 4,700 tests! It
must be reliable!"). But we don't want this sort of detail just yet, hence
my suggestion of a minimal spec.

Paul.