[Distutils] Question about testing

[P.J. Eby]

At 12:35 PM 8/17/2010 +0200, Ronald Oussoren wrote:
>Hi,
>
>What's the best way to bundle tests into a test archive for a 
>distribution using distribute or setuptools?
>
>My requirements:
>* Tests should not be installed because the testsuite is larger than 
>the actual package and not generally usefull
>* "python setup.py develop" should also not add the testsuite to sys.path
>* Some tests use extensions that are part of the testsuite
>* It should be possible to use "python setup.py test" to run the testsuite

With the exception of the "not add the testsuite to sys.path", these 
are trivially achieved by putting your test suite in a package 
directory alongside your code, and simply not listing that package in 
your packages to install.

Why do you not want the test suite on sys.path?  As long as you name 
it something like "mypackage_tests", it's unlikely to interfere with 
anything else.


>I also have a custom build_ext command to ensure that extensions in 
>the testsuite get copied there during the build phase
>(as if the inplace option is selected for just those extensions).

If you use the "setup.py test" command, the custom build_ext is 
unnecessary; the test command automatically does an in-place build_ext.

The one thing I'm not 100% positive on is whether you can build the 
test extensions without also installing them.  If not, then probably 
what you want is to subclass the *test* command instead, to have it 
add the test extensions, then fall through to the normal test command 
implementation.  That way, there's no possibility for them to be 
installed under normal conditions.


>This works but feels rather fragile, and I wonder if there is a 
>cleaner solution for meeting my requirements.
>
>(I'm using plain unittest for the tests and will probably switch to 
>unittest2 in the near future)

If you are using plain unittest, then making your tests a package and setting:

     test_suite = 'mypackage_tests',  # or whatever the package is called

will automatically test all TestCase subclasses in every submodule 
and subpackage, and also call any functions named 
"additional_tests()" to get extra test suites.

The latter is especially useful for running text-based doctests as 
part of your overall unittest suite; typically I'll have a few like this:

def additional_tests():
     import doctest
     return doctest.DocFileSuite(
         'README.txt', 'sometests.txt',
         optionflags=doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE,
     )

But of course you can return any sort of unittest.TestSuite.  The 
main point is just that if you have things in a module that you want 
to test that aren't just TestCase subclasses, you can slap them in an 
additional_tests() function for the "setup.py test" command to add in 
to the overall testing process.