[pypy-dev] Improved Unit Test Framework

[Peter Hansen]

holger krekel wrote:
> 
> [Anthony Baxter Sat, Oct 18, 2003 at 05:52:40PM +1000]
> >
> > >>> Laura Creighton wrote
> > > It might be nice to get a hold of other people and see what they
> > > would like to have in a unittest module.  That way, even if we
> > > don't write it that way ourselves, we can make sure that it will
> > > be easy for them to add their heart's desire later.
> >
> > It's probably worth breaking this into two categories:
> >
> >  - improvements to unittest itself

Just what would those be?  I still haven't heard a single useful
improvement.  There must be some (at least, that's obviously what
a lot of people feel), but what are they?

> >  - a test runner (e.g. Zope3's test.py, twisted's trial)

This was the area where we focused as well.  We needed the
recursive invocation feature, the change in structure (tests
in ./unit and ./accept subfolders rather than in same folder
as source), the automatic handling of .pth files on a per-project
basis, etc.  unittest.py was fine for us as-is, and we're even
using some bits exposed in its own TestRunner.

In the XP or TDD mailing list, "Phlip" reported on a system 
whereby merely by saving his files, his test suite would 
automatically be re-run and the results displayed in the test 
runner's window, and he could double-click on the errors and 
it would jump him directly back in the editor to the failing line.

> > The latter is what's _really_ needed - it'd be nice if something could be
> > done for Python 2.4.
> 
> Yes, the distinction makes sense.  ASFAIK most testframeworks provide a
> much better testrunner but don't care that much about the actual
> unittest machinery (because it's hard to change, anyway :-).  As a side
> note, i think that the distinction between unit-tests and functional
> tests is not always practical. We certainly do a lot of more-or-less
> functional testing in PyPy.  Actually, a new testing module shouldn't
> need to worry about unit<->function tests too much.

Agreed on that.  Although we separate the two for various reasons
into unit and acceptance test folders, in some cases its arguable which 
is which and in any case the testing framework doesn't distinguish at
all.  For us, the only thing that really distinguishes an acceptance
test, other than the inherent level of abstraction of the test, is the 
degree of mocking and "borging" that we have to do.  (At Kaval, a "borg" 
object is like a mock object in that it simulates the real thing, 
but in the case of the borg object it is a fake shell around the 
real code of something the program-under-test must talk to, such as 
a server.  Rather than run the server code on a real (say, Linux) server 
during the test, we will make a "borg" version that thinks it's on 
a real server but is just running in a separate process on the Win32 
test machine.)

-Peter