Unit testing data; What to test

[John J. Lee]

Sorry about the formatting etc. of this post -- I had to cut and paste
from Google.

> "John J. Lee" <phrxy at csv.warwick.ac.uk> wrote in message
> news:Pine.SOL.4.30.0102160309090.7729-100000 at mimosa.csv.warwick.ac.uk...
[...]
> If you keep the (stimuli, expected_responses) sets separate from the
> module being tested, you gain much the same benefits (and pay much
> the same costs) as by keeping documentation separate from code for
> other kinds of docs (think of tests as a form of executable docs...!).
>
> Keeping some tests/docs together with the code simplifies distribution
> issues: anybody who has your code also has this minimal set of tests
> (and other docs).

Well, that's not a problem, is it (unless the tests really are huge)?
They can be in the distribution without being mixed in with the code.

> However, in some cases, the total amount of tests
> and other docs can easily swap the amount of code -- and in this case
> keeping them together can easily be thought of as "too costly". The
> code can become practically unreadable if there are an order of
magnitude
> more lines of docs (including tests) than lines of actual source in a
.py.

Agreed.  My tests are separate from the code -- it hadn't occurred to me
to keep them in the same file.  It was the separation (or not) of test
data and test code I was wondering about, though.

> My favourite approach (and I do wish I was more systematical in
> practising what I preach:-) is to try and strike a balance: keep with
> the code (docstrings, comments, special-purpose test functions) a
> reasonable minimal amount of tests (& other docs) -- indicatively,
> roughly the same order of magnitude as the source code itself;
> but _also_ have a separate set of docs (& tests) for more extensive
> needs.
[...]

This is all good advice, but it doesn't actually answer my question.  ;-)
Anyway, I think I will try having some tests in the code, sounds like a
great idea considering the tests-as-docs aspect.

> > The second (third) one is vague: What / How should one test?  Discuss.
[...]
> > see, how do I justify this) it seems most of what is out there on the web
> > & USENET is either inappropriately formal, large-scale and
[...]
> > A few scattered XP
> > articles are the only exceptions I've found.
[...]
> If XP fits your needs, you could definitely do worse than adopt it
> wholesale!  Yes, much of what gets discussed about software
> development deals with large-scale SW (in testing and elsewhere) --
> that's because problems grow non-linearly with SW scale... when
> you release an application that's about 1,000 SLOC, it does not
> really matter much if your process and approach are jumbled; at

Let me assure you it _is_ possible to achieve an unmaintainable mess in
~1000 SLOC.  I have personally achieved this in, let's see (C-x C-f
munge.pl RET) ... 832 SLOC.  This is one of many things that Perl makes
easy (this is completely unfair to Perl of course -- what really makes it
easy is starting out thinking 'this is going to be a 100 line script' and
then extending by cut and paste -- I've learned my lesson, honest!).

> 10,000 SLOC, it's a problem; at 100,000, an utter nightmare, so
> you HAVE to be precise in defining your process then (or, call it

Yeah, of course, I understand that.  But there is still a place for lots
of testing for smaller efforts, minus the heavyweight formal process.

> 100, 1000, 10,000 FP -- but it's really about SLOC more than it
> is about FP, which is why higher-level languages help so much).
>
> Differently from what XP specifies, I think tests should be in two
> categories (and, similarly, so should deliverables be, rather than
> the just-one-deliverable which so characterizes XP -- that is a
> tad too X for my conservative self, even though I buy into 70+%
> of XP's tenets!).

Which two categories?  External and internal?  How is this different from
the XP unit test versus acceptance tests division?  Is your point just
that the external / internal division applies on more levels than just
final user / everything else -- which is what you seem to be talking about
below?

here's a snip explaining what XP people mean by acceptance tests for
anybody that hasn't read about it:

http://www.extremeprogramming.org/rules/functionaltests.html
----------

      Acceptance tests are created from user
stories. During an iteration the user stories selected
during the iteration planning meeting will be
translated into acceptance tests. The customer
specifies scenarios to test when a user story has
been correctly implemented. A story can have one
or many acceptance tests, what ever it takes to
ensure the functionality works.

      Acceptance tests are black box system tests.
Each acceptance test represents some expected
result from the system. Customers are responsible
for verifying the correctness of the acceptance tests
and reviewing test scores to decide which failed
tests are of highest priority. Acceptance tests are
also used as regression tests prior to a production
release.

----------

> Which, again, leads us to the internal/external
> tests-and-docs split.  External tests and docs (possibly, in large
> scale devt, on several scales: module aka unit, subsystem, whole
> system) deal with externals/interfaces (not just GUI's &c -- I'm
> talking about, e.g, module interfaces to other software; _of course_
> 'engine' and 'presentation' SHOULD almost invariably be separate
> components, but that's another plane of 'split').  Internal tests
> and docs deal with internals -- the kind of thing that needs to be
> tweaked at each refactoring.
>
> That's not the same dividing plane as the classic unit vs system
> test distinction -- it's slanted differently, and perks up again at
> each granularity level in a large-enough project (minimal granule
> being the module -- perhaps, at Python level, package -- not the
> single function or class, since functions and classes inside one
> module _are_ typically coupled too strongly to test/release/doc
> independently... there are exceptions, modules that are not very
> cohesive but rather collections of somewhat unrelated stuff for
> basically 'packaging' purposes, but they should be the exception
> rather than the rule).
[...]


John