[Tutor] Unit testing infinite loops

[Steven D'Aprano]

On Fri, Jan 31, 2014 at 11:31:49AM +0000, James Chapman wrote:
> Hello tutors
> 
> I've constructed an example which shows a problem I'm having testing a real
> world program and would like to run it past you.
[...]
> class Infinite_Loop_Tutor_Question(object):
>     def run_forever(self):
>         self.start_A()
>         time.sleep(0.5)
>         self.start_B()
>         try:
>             while True:
>                 time.sleep(1)
>         except KeyboardInterrupt:
>             print("Caught Keyboard Interrupt...")
>             sys.exit(0)
[...]
> In my example above, testing the everything but the run_forever method is
> trivial.
> 
> So on to my question... The run_forever method essentially just fires up a
> bunch of threads to serve various purposes and then waits for CTRL-C to
> terminate the entire program. Testing this at the moment is very difficult
> because the unit test ends up in the infinite loop. So, would a better idea
> be to create an attribute, set it to True and then do
> 
> try:
>     while self.attribute:
>         time.sleep(1)
> except KeyboardInterrupt:
>     ...


That probably won't hurt.

 
> My unit test could then set the attribute. However I'd still have the
> problem of how I get from the unit test line that fires up the method to
> the next line to change the attribute.
> 
> So how should the run_forever method be written so that it's testable, or
> if it's testable as is, how would I test it?

What are you trying to test? You don't just "test" a method, you test 
*something specific* about the method. So what specifically are you 
trying to test?


> And please, no comments about syntax, clean exits of threads, thread
> communication, resources, or even the need for testing the run_forever
> method. In my test I want to test that it makes the relevant calls and then
> enters the infinite loop at which point I want to terminate it.

Ah, you see, now you have a problem. Consider this function:

def long_calc():
    time.sleep(60*60*24*365)
    return 1


How do I test that the function returns 1? As given, I can't 
really, not unless I wait a whole year for the sleep() to return. So 
what I can do is split the function into two pieces:

def _sleep_a_year():
    time.sleep(60*60*24*365)

def _do_calculation():
    return 1

def long_calc():
    _sleep_a_year()
    return _do_calculation()


Now I can unit-test the _do_calculation function, and long_calc() is now 
simple enough that I don't really need to unit-test it. (Unit testing 
should not be treated as a religion. You test what you can. Any testing 
is better than nothing, and if there are some parts of the program which 
are too hard to test automatically, don't test them automatically.)

Or, I can monkey-patch the time.sleep function. Before running my 
test_long_calc unit-test, I do this:

import time
time.sleep = lambda n: None

and then restore it when I'm done. But like all monkey-patching, that's 
risky -- what if the calculation relies on time.sleep somewhere else? 
(Perhaps it calls a function, which calls another function in a module 
somewhere, which calls a third module, which needs time.sleep.) So 
monkey-patching should be a last resort.

Another alternative is to write the function so it can be 
tested using a mock:

def long_calc(sleeper=time.sleep):
    sleeper(60*60*24*365)
    return 1


Then I can test it like this:

assert stupid(lambda n: None) == 1

where the lambda acts as a mock-up for the real sleep function.


Let's look at your method. As given, it's too hard to test. Maybe you 
could write a unit test which fires off another thread, which then 
sleeps for a few seconds before (somehow!) sending a KeyboardInterrupt 
to the main thread. But that's hard to explain and harder to do, and I 
really wouldn't want to rely on something so fiddly. So let's re-design 
the method with testing in mind.

First, pull out the part that does the infinite loop:

    def do_infinite_loop():
        # Loop forever. Sleep a bit to avoid hogging the CPU.
        while True:
            time.sleep(1)


That's *so simple* that it doesn't need a test. The body of the method 
is two short, easy lines, plus a comment. If somebody can read that and 
be unsure whether or not it works correctly, they're in trouble.

But, if you like, you can make it more complicated. Have the method 
check for a magic global variable, or a instance attribute, or 
something:

    def do_infinite_loop():
        # Loop forever. Sleep a bit to avoid hogging the CPU.
        # Perhaps not forever.
        if hasattr(self, 'DONT_LOOP_FOREVER'):
            x = 10
            while x > 0:
                time.sleep(1)
                x -= 1
        else:
            while True:
                time.sleep(1)


Yuck. Now you have added enough complication that it is no longer 
obvious that the method works, and while you can test the non-infinite 
loop part, you still can't test the infinite loop part. No, better to 
stick with the simplest thing that works.

Now for the rest of your method:


    def run_forever(self):
        self.start_A()
        time.sleep(0.5)
        self.start_B()
        try:
            self.do_infinite_loop()
        except KeyboardInterrupt:
            print("Caught Keyboard Interrupt...")
            sys.exit(0)


How does this help us with testing? We can monkey-patch the 
do_infinite_loop method!


class MyTest(unittest.TestCase):
    def test_run_forever_catches_KeyboardInterrupt_and_exits(self):
        def mock_looper(self):
            raise KeyboardInterrupt
        instance = Infinite_Loop_Tutor_Question()
        # Monkey-patch.
        instance.do_infinite_loop = mock_looper
        self.assertRaises(SystemExit, instance.do_infinite_loop)


Testing that it prints the appropriate message, I leave as an exercise. 
(Hint: you can re-direct stdout, run the method inside a try...except 
block, then restore stdout.)

(By the way, I haven't tested any of this code.)


-- 
Steven