Quote of the day

[Steven D'Aprano]

On Thu, 19 May 2016 02:05 am, Ethan Furman wrote:

> On 05/18/2016 08:35 AM, Thomas Mlynarczyk wrote:
>> On 18/05/16 17:21, Ned Batchelder wrote:
> 
>>> Ideally, an empty test wouldn't be a success, but I'm not sure how
>>> the test runner could determine that it was empty.  I guess it could
>>> introspect the test function to see if it had any real code in it,
>>> but I don't know of a test runner that does that.
>>
>> Simple: a function which does not produce at least one "failure" or
>> "pass" does not test anything. No need to introspect the code. Just
>> check if the total score of failures and passes has changed after the
>> function was run.

I think you have misunderstood how unittest currently works. A do-nothing
test already counts as a pass. Here's a dumb test which is pointless,
followed by an even dumber one that does literally nothing:


[steve at ando ~]$ cat dumbtest.py
import unittest

class MyTest(unittest.TestCase):
    def test_something(self):
        self.assertEqual(100, 100.0)
    def test_nothing(self):
        pass


And now watch as both the pointless and the do-nothing tests count as
passed:


[steve at ando ~]$ python -m unittest --verbose dumbtest
test_nothing (dumbtest.MyTest) ... ok
test_something (dumbtest.MyTest) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK



So to start with, for your solution to be workable, we'd have to change the
way unittest decides what is a success and what isn't.

 
> Not so simple: I have tests that do nothing besides build objects.  If
> building the objects raises no errors the test passed.

It wouldn't be hard to add a "success" method to unittest, so that after
building the object you just call self.success() to flag it as passing.

But now the obvious way to have fake unittests is:

    def test_nothing(self):
        self.success()


The problem here is not a technical problem. It is a cultural or human
problem: somebody, due to malice, incompetence, overwork, ignorance or
stupidity, wrote a fake test that didn't actually test anything. Maybe
their intentions were good, and they meant for it to do something and it
just got forgotten... or maybe they deliberately thought that they could
satisfy the letter of the requirement "must have unit tests" without
putting in the hard work to satisfy the spirit of it.

Either way, until the testing framework contains enough artificial
intelligence to actually reason about whether the test is *useful* or not,
there's no technological way to solve this problem. You need a person[1]
intelligent enough to make a judgement "wait a minute, this code doesn't
test anything useful".

And that's a hard problem. Even human beings do poorly at that. The idea
that the test framework could solve it is naive.


> Although, for the benefit of empty tests not passing I could add a
> do-nothing assert:
> 
>    self.assertTrue(created_obj)
> 
> (it's a do-nothing because if the object wasn't created the test would
> have already failed).

It wouldn't have failed. It would have raised an exception, which is
different. (Curse you English, we need more words for describing kinds of
failure!!!)

unittest supports four different test results:

- pass
- fail
- error (raise an exception)
- skip

which print as . F E S respectively. The tests you're describing will print
as E rather than F, which is better than a failure. A test failure is code
that silently does the wrong thing:

    "I find it amusing when novice programmers believe their
     main job is preventing programs from crashing. ... More
     experienced programmers realize that correct code is
     great, code that crashes could use improvement, but
     incorrect code that doesn’t crash is a horrible nightmare."
     -- Chris Smith

while an E means that your code will cleverly raise an exception instead of
doing the wrong thing :-)






[1] Human or machine person, I don't care.


-- 
Steven