[Distutils] Distribute testing

[Tarek Ziadé]

On Sat, Dec 19, 2009 at 6:33 PM, ssteinerX at gmail.com
<ssteinerx at gmail.com> wrote:
[..]
>
>        While this approach will be fine when we move it to building on-demand buildbots, it is much too time consuming both in development time and real-time processing to use for this release cycle.  It would be much better (for now) to just make sure you're running it on a machine that can do the job.

*or* to update an existing test environment. e.g. check if python !=
trunk is already present.


[..]
>        First, we need to decide where to put the testing sub-project.  I'm asuming it can just live in the /tests subdirectory, maybe with a subdir to hold this whole project as its own module.


Sounds good

>
>        Here's what I'd like to see it do:
>
>        Run pre-test checks i.e. verify that the machine is capable of running the tests:
>
>                1>      Assume that `python2.4`, `python2.5`, `python2.6`, `python3.1` will invoke a properly   set up Python interpreter.  This is beyond the scope of this project; might be taken up at a later time.
>
>                2>      Create a simple way to verify installed utilities before proceeding with tests (anyone have one of these around?).
>                Something like:
>                        ASSERT(`nosetests -V` == `nosetests version 0.11.1`)
>
>                3>      Allow a certain amount of corrective action to be able to be taken to satisfy the assertions above (like, install nose version 0.1.11 from PyPi, for example) for each supported version of Python.
>
>        Then:
>                for each product to be tested:
>                        for each Python version this product is to be tested on:
>                                install the product
>                                run its test suite
>                                record any failures
>
>        Pretty simple, but will avoid any of the sorts of issues we just had with the most recent
> release, is very easy to add new products to (just add a configuration section), and can be easily
> moved out to the buildbots when it does what we want.

Sounds good too ! Notice that we also need to try those tests in
various environments, besides
python versions: virtualenv, no virtualenv, setuptools present before
we start, not present, etc

Regards,
Tarek