best python unit testing framwork

[James Harris]

On 15 Nov, 01:02, "Brendan Miller" <catph... at catphive.net> wrote:
> On Thu, Nov 13, 2008 at 3:54 AM, James Harris
>
>
>
> <james.harri... at googlemail.com> wrote:
> > On 11 Nov, 22:59, Brendan Miller <catph... at catphive.net> wrote:
> >> What would heavy python unit testers say is the best framework?
>
> >> I've seen a few mentions that maybe the built in unittest framework
> >> isn't that great. I've heard a couple of good things about py.test and
> >> nose. Are there other options? Is there any kind of concensus about
> >> the best, or at least how they stack up to each other?
>
> > You don't mention what you want from testing so it's hard to say which
> > is "best" as it depends on one's point of view.
>
> > For example, I had a requirement to test more than just Python
> > programs. I wanted to test code written in any language. None of the
> > frameworks I found supported this so I wrote my own tester. It
> > interacts with programs via file streams - principally stdin, stdout
> > and stderr though others can be added as needed.
>
> I was speaking to unit tests (tests of individual classes and
> functions in isolation of the rest of the program) and a test driven
> development approach. I find those to be much more useful, and good
> for pinpointing bugs and guiding development. In contrast regression
> tests tend to be slow, and are the programmatic equivalent of kicking
> the tires and seeing if they fall off.

I agree with your points but would add

1. We should write smaller modules than we generally do. It promotes
clear thinking and encourages the production of reusable code. In
Python it is easy to add a __main__ section to interface to the
classes or functions defined in the code. Such code can then be tested
as mentioned. The functions or classes can still be imported into and
used in other code as needed.

2. The example given wasn't intended to be the only way forward but to
highlight that what makes a test package good depends on your
requirements (which I don't think you mentioned). In my case the
desire to test arbitrary languages was a killer blow to various Python-
only frameworks which I wasn't too impressed with anyway. (At least I
saw none that looked worth investing the time to learn.) Similarly an
option for you is to write your own tester. It may be a better fit
than someone else's package and not too hard to do. On the other hand
a pre-written package may be better - depending on what you are
looking for.


> I've also seen regression tests lead to a "sweep the bugs under the
> rug mentality" where developers will code to prevent errors from
> crashing the regression test, e.g. by catching and swallowing all
> exceptions without fixing the underlying problem. It's easy to fool
> regression tests since what it does works at such a high level that
> most aspects of program correctness can't be directly checked. This is
> very frustrating to me because it actually leads to lower code
> quality.


If you have dodgy developers then much is wrong, not just testing. If
the team works well, IMHO, production of test cases should be a
welcome part of the development process. When is it better to find
bugs: before or after shipping to a client? Test cases can be put
together in various ways. For example, if one has the code and chooses
to work this way white-box testing can lead to generation of test
cases. On the other hand if an organisation prefers it can produce
test cases from the spec - or they can do both. It's their choice but
in any case the developers and testers should work together.


> > One nice by-product is that test code does not bloat-out the original
> > source which remains unchanged.
>
> That's the main reason most people don't write unit tests. It forces
> them to properly decouple their code so that parts can be used
> independently of one another. Adding such things to messy ball of mud
> code after the fact is an enourmous pain in the butt. Thankfully,
> since python is duck typed and doesn't require lots of boilerplate
> writing interfaces and abstract factories (since the class object
> itself acts as an abstract factory), writing properly decoupled code
> in the first place doesn't look nearly as hard as in C++ or Java.

You could still import one Python module into another to run test
cases such as

-----
import test_support
import <program_to_be_tested>

<tests of program_to_be_tested using test_support>
-----

where the tests can interact directly with the components of the code
under test. The test support code can handle things such as collating
results and reporting as well as providing code to directly support
certain tests, handle exceptions etc.

I wouldn't be surprised if good testing code were in many cases to be
significantly longer than the code being tested. Keeping the tests in
a separate file keeps the code being tested shorter which can only
improve clarity. It also allows either the developer or someone else
to write the tests. Putting code and testing code in the same file
almost requires one person to write both parts.

A final thought: if you have dodgy developers who engineer their code
to avoid tests you're not going to have much success in getting them
to write good test code (though, hopefully, the mental process of
writing test cases should help them improve the code they write).

At the end of the day all I'm saying is that the best test framework
for you really depends on your requirements, rather than what others
think is "best" or not.

--
James