[issue12250] regrtest: make --timeout explicit

[Stefan Krah]

Stefan Krah <stefan-usenet at bytereef.org> added the comment:

STINNER Victor <report at bugs.python.org> wrote:
> > The implicit timeout in regrtest.py makes it harder to write automated
> > test scripts for 3rd party modules. First, you have to remember to
> > set --timeout=0 for long running tests.
>
> Ah? Which test is too long for the default timeout? On which kind of computer?

I don't know about the regular tests, though I think it might be possible
to exceed the timeout if you run the tests on a very slow machine under
Valgrind.


For cdecimal, I've integrated a facility into test_decimal.py to test
all failures of Python API functions in a systematic manner. This
works by wrapping all test_* methods in a function that first counts
the number of API calls in a test case, then reruns the test case
n times with a deliberate failure point moved ahead each time.

This facility is of course not enabled by default and requires a small
patch to _decimal.c. When enabled, run times easily exceed 1 hour,
especially with Valgrind.

The broader point is that due to the excellent refleak counting
facility people might use (I hope!) regrtest.py for 3rd party
modules, where we don't know the run times of the tests.


> > Then, you have to remember not to use the --timeout option when
> > compiling --without-threads.
>
> We can change regrtest to just print a warning, and not exit, if Python has no
> thread support and --timeout option is used.

This would already help a lot. Currently I have to do this in my test
scripts:

    timeout="--timeout=0"
    if [ X"$args" = X"--without-threads" ]; then
        timeout=""
    fi



> Well, the first solution (display a warning in regrtest, but don't exit) looks
> simpler :-)

Indeed. :)


> > I'd much prefer that there's no timeout unless explicitly specified.
> > For the buildbots, I think this could be done in the Makefile.
>
> I set a default timeout for buildbots, because I don't know how to change how
> regrtest is started on all buildbots, but you are right: it can be done in
> "make buildbottests".

This would be my favorite solution. Of course other people might like the
default timeout - it's hard to satisfy everyone.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12250>
_______________________________________