[issue24845] IDLE functional/integration testing

[Terry J. Reedy]

Terry J. Reedy added the comment:

A different take on the proposal: A functional integration test for IDLE should open IDLE, rename .idlerc, open a new file, insert some text, and, for instance, at some point open the options dialog for testing.  As part this, the font face change would simulate button clicks and then check that the the font had changed in the editor *and* in .idlerc/config-main.cfg.  I have done this manually.  A complete functional test would do what I also occasionally do.  Exercise at least once every menu function, all dialog widgets, and a few invisible functions.  Unless suppressed, even the print function should be included, verified by the user affirming that the printed page appeared.

Such a test should be separate from the unittest test suite, just as is the htest module.  Only a few buildbots run gui tests anyway, but some core devs run the suite regularly, some with gui enabled.  I don't want to impose on them in any way, neither another minute or two of test time, nor flickers and bells.  Running such a test suite should be strictly voluntary.

A few files could go in idle_test, like htest.py.  A large number might justify a separate func_test directory.

I don't want the mode of invocation and operation and the code style to be limited to that of unittest.

I envision IDLE and the functest code running in one process, with one Tk instance.  Which starts first is to be decided.  But if IDLE starts first, then Run Func Test can be a menu item.  I have already though of adding Run Unit Tests (in a subprocess) and Run Visual Tests (in the same process) to the Help Menu. 

The main use of setup and teardown in IDLE unit tests is to create and destroy Tk roots;  with one permanent root, these functionsgo away.

If test_xyz functions are defined at module scope, they are easy to collect and run.  This is the function I use in my own project.

def main(namespace):
    for name, obj in namespace.items():
        if name.startswith('test_') and hasattr(obj, '__call__'):
            print(name)
            obj()

The obj() call could be wrapped with

 try: obj()
 except Exception as e: print(e, file=output_window)

where output_window is an instance of IDLE's OutputWindow.  A helper equal(x,y) function could send an error message to the same place.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue24845>
_______________________________________