[Python-Dev] sprints and pushes

[Nick Coghlan]

On Fri, Mar 25, 2011 at 12:51 PM, Stephen J. Turnbull
<stephen at xemacs.org> wrote:
> Eg, much as I normally respect Barry's intuitions, his proposal (to
> remove costly tests, without reference to the possibility of missing
> something important) is IMHO absolutely the wrong criterion.  I don't
> really know about Python's test suite, but in XEmacs the time-
> expensive tests are mostly the ones that involve timeouts and lots of
> system calls because they interact with the OS -- and those are
> precisely the finicky areas where a small change can occasion an
> unexpected bug.  For XEmacs, those bugs also are more likely than
> average to be showstoppers for dependents, in the sense that until
> they're fixed, the dependents can't be tested at all.  So the cost of
> *not* running those tests is relatively high, too.

For Python, our most expensive, slow tests are generally process
related or IO related (over time the threading related ones have been
largely fixed to use Event based signalling rather than relying on
timeouts, so they're significantly faster than they once were).

These slow tests are *also* typically the most platform dependent
tests, so a "green light" from a local test run is generally pretty
inconclusive - you don't really find out whether you borked something
until you get green lights on the buildbots for platforms other than
those the patch was developed on.

So I still see value in having a standard "smoke test" that runs
through the platform independent stuff, to reduce the number of minor
errors that needlessly cause the buildbots to go red.

The idea would be to create a tiered test suite along the following lines:

1. The buildbots: run the entire (-uall) test suite across a variety
of platforms. Performed for every commit pushed to
hg.python.org/cpython.
2. Complete local test: run the entire (-uall) test suite on a local
working copy. Recommended before first committing a fix or change to a
branch.
3. Basic local test: run the test suite with no additional resources
enabled on a local working copy. Current closest equivalent to a
"smoke test"
4. Proposed "smoke test": quick test of platform independent code for
use when merging heads after a push race
5. Specific tests: run specified tests for modules being worked on.
Used during development to check fix validity and feature degree of
completion.

With the volume of platform dependent code we have in CPython and the
standard library, the only way we're ever likely to achieve
consistently green buildbots is to move to a "staging repo" model,
where commits only get made to the central repo after they have
already passed muster on the buildbots at least once. I think that's
actually a good way for us to go in the long run, but I also think
separating out a fast set of platform independent is a decent idea.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia