[issue33873] False positives when running leak tests with -R 1:1

[STINNER Victor]

STINNER Victor <vstinner at redhat.com> added the comment:

I tested PR 7735:

vstinner at apu$ ./python -m test -R 0:0 test_os -m test_access
Run tests sequentially
0:00:00 load avg: 0.19 [1/1] test_os
beginning 0 repetitions


test_os leaked [] references, sum=0
test_os leaked [] memory blocks, sum=0
test_os failed

== Tests result: FAILURE ==

1 test failed:
    test_os

Total duration: 63 ms
Tests result: FAILURE

vstinner at apu$ ./python -m test -R 0:1 test_os -m test_access
Run tests sequentially
0:00:00 load avg: 0.35 [1/1] test_os
beginning 1 repetitions
1
.
test_os leaked [280435] references, sum=280435
test_os leaked [91518] memory blocks, sum=91518
test_os leaked [4] file descriptors, sum=4
test_os failed

== Tests result: FAILURE ==

1 test failed:
    test_os

Total duration: 95 ms
Tests result: FAILURE

vstinner at apu$ ./python -m test -R 1:0 test_os -m test_access
Run tests sequentially
0:00:00 load avg: 0.16 [1/1] test_os
beginning 1 repetitions
1
.
test_os leaked [] references, sum=0
test_os leaked [] memory blocks, sum=0
test_os failed

== Tests result: FAILURE ==

1 test failed:
    test_os

Total duration: 95 ms
Tests result: FAILURE



Hum, we should require at least one run and at least one warmup: -R 1:1 should be the bare minimum.

By the way, it seems like negative numbers are currently accepted, whereas it doesn't make sense:

vstinner at apu$ ./python -m test -R 0:-2 test_list
Run tests sequentially
0:00:00 load avg: 0.31 [1/1] test_list
beginning -2 repetitions
(...)

It would fix this bug as well.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33873>
_______________________________________