[Python-checkins] devguide: Issue #18871: make it more explicit that the test suite should be run before

[Terry Reedy]

On 8/28/2013 5:51 PM, antoine.pitrou wrote:

> +Does the test suite still pass?

There are several assumptions packed in that question.

0. The test suite is relevant to the patch.

Not true for doc patches, some idlelib patches, and probably some others.

1. The test suite runs to completion.

For at least a couple of years, that was not true on Windows desktops. 
The popup Runtime Error boxes from now are now gone, but a month ago, a 
crashing test froze Command Prompt.

2. The test suite runs without error.

I have hardly ever seen this on Windows, ever (with sporadic runs over 
several years). Today, test_email and test_sax failed in successive 
runs*. Tomorrow, who knows?

3. If there is a new failure, it is due to the patch.

There have been at least one, but I think more, intermittent failures of 
Windows tests in the last few months

So the real question is: "Does the patch cause a new (and 
non-intermittent) failure?" Answering this is not cheap.

4. The gain of answering that question is worth the cost.

I think it depends on the patch and the apriori expectation.


* By default, Windows Command Prompt does not hold a complete run, let 
alone two, so one would have to remember or write down what failed 
before. Increasing the number of lines pads the window with blank lines, 
making scrolling up and back down much harder. A new Idle feature to run 
the suite with a choice of options and put the results in a new Output 
Window would make full-suite testing much easier, at least on Windows.

> +.. note::
> +   You really need to run the **entire** test suite.  Running a single test
> +   is not enough as your changes may have unforeseen effects on other tests
> +   or library modules.

I worry that further emphasizing an overly broad, time-consuming, and 
sometimes impractical rule will only discourage more participation.

> +   Running the entire test suite doesn't guarantee that your changes
> +   will pass the :ref:`continuous integration <buildbots>` tests, as those
> +   will exercise more possibilities still (such as different platforms or
> +   build options).  But it will at least catch non-build specific,
> +   non-platform specific errors, therefore minimizing the chance for
> +   breakage.

Running the suite in any particular order misses a probabilistic half of 
order-dependent errors. An option to run a target test both first and 
last, so it runs both before and after all other tests, would catch most 
of the rest.


--
Terry Jan Reedy