Unit-testing single function with large number of different inputs

[Robert Ferrell]

I'm not a unit-testing expert, and I don't necessarily recommend what
I'm about to suggest, but...

To solve the problem of having a single "test" actually run many
"internal" tests, report on the results of those many tests, and fail
(in the PyUnit sense) if any of those internal tests fail, you could
add a flag, AllTestsPassed, to the TestLoad and TestValidate classes. 
Initialize that flag to True.  Then, in your version of assertEqual,
if any test fails set the flag to False.  Finally, after you've run
all your tests, add

  self.failUnless(self.AllTestsPassed, 'Some tests failed.')

HTH,

-robert
Edvard Majakari <edvard+news at majakari.net> wrote in message news:<87ptcdsyyt.fsf at titan.staselog.com>...
> Hi all ya unit-testing experts there :)
> 
> Code I'm working on has to parse large and complex files and detect
> equally complex and large amount of errors before the contents of the file
> is fed to module interpreting it. First I created a unit-test class named
> TestLoad which loaded, say, 40 files of which about 10 are correct and
> other 30 files contained over 20 different types of errors. Different
> methods on the TestLoad class were coded so that they expected the
> spesific error to occur. They looked like the following:
> 
> def test_xxx_err(self):
>     for fname in xxx_err_lst:
>        self.assertEqual(get_load_status(fname), xxx)
> 
> However, now we've added several more tests and we have added another
> function to the system - the ability to try loading the file without
> actually executing instructions in it. So, I added another class
> TestValidate, which has exactly the same tests as the TestLoad class, the
> only difference being in the routine get_load_status, which is not
> get_load_status but get_validate_status.
> 
> I thought I'd simplify the process of adding new tests - if it is too
> cumbersome to add new tests, we'll end up with having few tests which in
> turn will result in more buggy software system. 
> 
> Thus I came up with the simple idea of adding two directories:
> test/correct-files and test/incorrect-files. Every time the test suite is
> run, it goes through every file in the test/correct-files and
> test/incorrect-files and expects to receive status 0 from all files under
> correct-files, and appropriate error code from all files under
> test/incorrect-files. How does it deduce the correct error code? Well,
> I've forced things to that any file in the test/incorrect-files must
> begin with code prefix 'xx-' in the file name, and the test suite reads
> expected error code from the file name prefix. 
> 
> All this allows me to have only two methods (in addition to setUp and
> tearDown methods) in TestLoad and TestValidate classes, and what's more
> convenient - I don't have to alter unit test suite at all when I add new
> file tests to the system; all I have to do is to put it under either
> directory, and if the file contains an error, I prefix the file name with
> appropriate code.
> 
> The system has but one annoyance: first, if I use self.assertEqual in the
> test_load method, it aborts on the first error found and as such doesn't
> test all the other files. This would be bad, because when errors occur in
> the code, I have to fix them one by one running test suite after every fix
> to find other potential errors in the system. 
> 
> So I changed the test_load routine by replacing self.assertEqual with my
> own code which won't abort the test if errors are found, but it will
> report them with sufficient verbosity and continue testing. But this
> approach isn't optimal either: because there is no self.assertEqual in the
> test anymore, there may be many errors in the test class but after running
> all the methods, the test suite will report OK status, even though errors
> did occur.
> 
> The problem I described above shows probably one of the best reasons why
> each test method should test only one (type of) thing, but in my special
> case I don't find that appropriate due to aforementioned reasons. Any
> suggestions? I was wondering about creating test methods dynamically
> according to the files found in the test directories (which would solve
> all the problems), but I find it a tad too complex way of solving the
> problem.