[issue25220] Enhance and refactor test.regrtest (convert regrtest.py to a package)

[STINNER Victor]

New submission from STINNER Victor:

The Lib/test/regrtest.py file became a monster. It's very long, it uses a lot of global variables. It's hard to review and hard to maintain. I propose to:

- split single file regrtest.py into multiple files in a new Lib/test/regrtest/ package
- refactor the test runner as a class to avoid global variables
- refactor the multiprocessing test runner as a class in a separated file

Attached patch implements this idea.


The following commands still work:

   ./python -m test [options] [args]
   ./python -m test.regrtest [options] [args]

But the following command doesn't work anymore:

   ./python Lib/test/regrtest.py [options] [args]

If regrtest.py is executed directly on buildbots, these buildbots should be modified first.


The "./python -m test.regrtest" command should work on all Python versions (Python 2 and Python 3) and so should be preferred. The "./python -m test" command only works on Python 3 (thanks to the new __main__.py feature).


The change adds a new feature: it now displays name of concurrent tests running since 30 seconds or longer when the multiprocessing test runner is used (-j command line option). Example:

    ...
    [240/399] test_uu
    [241/399] test_urllib_response
    [242/399] test_struct
    [243/399] test_descrtut
    [244/399] test_threadedtempfile
    [245/399] test_tracemalloc -- running: test_concurrent_futures (30 sec)
    [246/399] test_dbm_dumb -- running: test_concurrent_futures (30 sec)
    [247/399] test_codeop -- running: test_concurrent_futures (30 sec)
    ...
    [395/399/1] test_asyncio -- running: test_multiprocessing_fork (40 sec), test_multiprocessing_spawn (44 sec)
    [396/399/1] test_faulthandler -- running: test_multiprocessing_fork (50 sec), test_multiprocessing_spawn (54 sec)
    [397/399/1] test_multiprocessing_fork (52 sec) -- running: test_multiprocessing_spawn (56 sec)
    [398/399/1] test_multiprocessing_spawn (68 sec) -- running: test_multiprocessing_forkserver (39 sec)
    [399/399/1] test_multiprocessing_forkserver (50 sec)

I want this feature to analysis why more and more buildbots fail with a timeout without saying which test was running (well, I suspect multiprocessing tests...).

Note: faulthandler can show where regrtest is blocked, but not when the multiprocessing test runner is used. And sometimes the process is killed by the buildbot, not by faulthandler :-/


Another minor new feature: on CTRL+c, it shows which tests are running when the multiprocessing test runner is used. Example:

    [ 38/399] test_dummy_thread
    [ 39/399] test_codecmaps_jp
    [ 40/399] test_future5
    ^C
    Waiting for test_scope, test_decimal, test_memoryview, test_heapq, test_unicodedata, test_trace, test_threadsignals, test_cgitb, test_runpy, test_cmd_line_script


Other changes:

* Show test timing when a test runs longer than 30 seconds

* Don't make __file__ absolute, findtestdir() calls os.path.abspath() instead. Remove these lines:

    __file__ = os.path.abspath(__file__)
    assert __file__ == os.path.abspath(sys.argv[0])

* print() is now called wih flush=True (it didn't check all calls, only the major calls), remove sys.stdout.flush() and  sys.stdout.flush()

* A lot of refactoring. Sorry, I didn't take notes for each change.


I fear that test_regrtest has a small code coverage. I only tested major options, I didn't test -R for example.


Note: I don't understand how the --single option works when regrtest is not run from the Python source code directory. A temporary directory, so the pynexttest file is removed after its creation, no? If it doesn't make sense to use --single outside Python directory, maybe an error should be raised?

----------
components: Tests
files: regrtest_package.patch
keywords: patch
messages: 251419
nosy: ezio.melotti, haypo, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Enhance and refactor test.regrtest (convert regrtest.py to a package)
versions: Python 3.6
Added file: http://bugs.python.org/file40554/regrtest_package.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25220>
_______________________________________