unittest: Calling tests in liner number order

[Diez B. Roggisch]

> Here's an example of why *running* tests in order can make sense.
> 
> You could have a bunch of tests of increasing complexity.  The first bunch 
> of tests all run in a few seconds and test some basic functionality.  From 
> experience, you also know that these are the tests that are most likely to 
> fail as you port to a new environment.
> 
> There's also some tests which take a long time to run.  If the basic stuff 
> that's being tested by the earlier tests doesn't work, there's no way these 
> tests could pass, but they still take a long time to fail.
> 
> It's really handy to have the simple tests RUN first.  If you see they 
> fail, you can cancel the rest of the test run and get on with fixing your 
> code faster.

I don't see this as something that can be solved by ordering tests - 
*especially* not on a per-method-level as the OP suggested, because I 
tend to have test suites that span several files.

Instead when I've been in the situation that you describe before, I 
resorted to an approach where I annotated tests as long or short-running 
(or any other criteria), and then ran the tests that were appropriate.

For example, post-commit-tests needed to be short, as otherwise the 
feedback came to slow.

  Annotation could be done explicit, or implicit by grouping tests 
together that had the desired property. However, as I prefer tests that 
share e.g. the same module to test the same functionality, I rather have 
an annotation mechanism.

So *if* anything should be changed IMHO it would be the introduction of 
a tagging system or something equivalent, together with a selection 
mechanism based on that.

> It's a good thing to make it easy to do things the right way, and difficult 
> to do things the wrong way.  The danger is when you let your pre-conceived 
> notions of right and wrong trick you into making it difficult to do things 
> any way but YOUR way.
> 
> So far, the strongest argument I've seen against the OP's idea is that it's 
> not portable to Iron Python.  That's a legitimate argument.  All the rest 
> of the "You're not supposed to do it that way" arguments are just religion.

So far the reasons for introducing them haven't been compelling either. 
Neither does it work over several testsuites, nor is it the only thing 
that can order *results*, which you (rightly so) claimed as being useful.

Diez