[Python-Dev] setUpClass and setUpModule in unittest

[Holger Krekel]

On Fri, Feb 12, 2010 at 12:00 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On 2010-02-11 16:20 PM, Ben Finney wrote:
>>
>> Guido van Rossum<guido at python.org>  writes:
>
>>> The argument that a unittest framework shouldn't be "abused" for
>>> regression tests (or integration tests, or whatever) is also bizarre
>>> to my mind. Surely if a testing framework applies to multiple kinds of
>>> testing that's a good thing, not something to be frowned upon?
>>
>> To my mind, an API should take a stand on the “right” way to use it,
>> rather than being a kitchen-sink of whatever ideas had any support.
>> Doing something the right way should be easy, and doing something the
>> wrong way should be awkward.
>
> setUpClass and setUpModule are the "right" way to do many types of
> integration and functional tests. Integration and functional tests are vital
> tasks to perform, and unittest provides a good framework otherwise for
> implementing such tests.

Ben just expressed his opinion about API design and you claim some
truth about testing in general.  In my experience, integration and
functional testing is a  complex and evolving topic, usually requiring
more from the tool or framework than classic unit-testing. To name a
few issues:

* creating tempdirs and files
* setting up base environments
* starting and stopping servers
* mocking components
* replaying individual tests
* reacting to timeouts
* handling asynchronicity
* web-javascript integration support
* configuring fixtures from config files
* CI tool integration and multi-platform deployment
* running variations of the same tests across different base configs
* ... much much more

It's true that you can go and extend unittest for that but a) unittest
is just a tiny bit of what is involved for satisfying the needs  b)
what you are doing then is mostly using the fact that a setup function
(or chain) is invoked and a test function is invoked and that python
has some builtin modules for handling the above issues.   And you are
using Python - and Python is nice and (potentially) concise for
writing tests, sure.  That's not wholly the fault of the unittest
module, though :)

So. Doing fixtures via static encoding in class and module setup
functions is a way to provide a generic framing for writing tests.
The "right" way?  In many cases and for the about 6 different people i
interacted with (and on actual RL code) in the last 2 weeks it does
not help incredibly much.  There is experiences from other test tool
authors indicating similar experiences.

I will say that module/class can be helpful and you can do some useful
things with it and thus it makes some sense to add it for std-unittest
but claiming this is great and most of what you need for "many types"
of functional testing is misleading and plays down the many good
things you can do with Testing and Python.

best,
holger