[Python-Dev] Trimming the fat from "make quicktest" (was Re: I am now lost - committed, pulled, merged, what is "collapse"?)

[Ethan Furman]

skip at pobox.com wrote:
>     Barry> Not if you keep in mind the appropriate use case for each of the
>     Barry> separate make test targets.
> 
> Programmers are lazy.  They will often take the shortest path.  Fix a small
> bug in module X which seems innocent enough, fail to recognize that it
> breaks module Y.  Run "make smoketest" and see that all the test_X stuff
> passes.  Don't notice that key test_Y tests are not even run, push, then
> head out to lunch.  Come back to (hopefully) a bunch of red buildbots.
> Still, it would have been good to catch that problem before heading out to
> Buffalo Wild Wings to watch football players trip over sprinkler heads.
> 
> How many of us really and truly can't wait a few minutes for the test suite
> to complete?  Especially once Antoine (or whoever) gets -j working properly.

I think the use-case has been lost.  Think sprints and multiple push 
races.  No one is arguing that the smoke-test should be the default, but 
seriously, are you willing to spend an hour or more re-running the 
complete suite of tests six, eight, or 12 times because of push races in 
a sprint?  I can see losing a good portion of your sprinting day.

Which tests are included in the smoketest definitely needs careful 
reviewing.

Perhaps a better solution for sprints is to clone, have the sprinters 
clone from that clone, have one person responsible for running full 
tests, have the others push to the sub-sub-clone.

~Ethan~