[issue44708] Regression tests with -w should only re-run affected test methods, not the entire file

[Łukasz Langa]

New submission from Łukasz Langa <lukasz at langa.pl>:

When the CPython test suite re-runs a flaky failed test, it doesn’t actually re-run just that failed test but the entire test file. The most common case for flaky tests is when networking, threading, or multiprocessing is involved. Frustratingly those are some of the largest test files we have. Re-running them takes a lot of time.

Instead of re-running the entire file, regrtest should only re-run what actually failed.

NOTE: I added 3.10 and 3.9 to this issue even though it's not a pure bugfix. The reason is that this can visibly speed up CI for open pull requests.

(Of course, it would be best to avoid flaky tests altogether and we’re working on that, but re-running what failed is a pragmatic stopgap that is necessary in times of distributed CI that we don’t fully control.)

----------
assignee: lukasz.langa
components: Tests
messages: 397985
nosy: lukasz.langa, pablogsal
priority: normal
severity: normal
stage: patch review
status: open
title: Regression tests with -w should only re-run affected test methods, not the entire file
type: resource usage
versions: Python 3.10, Python 3.11, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue44708>
_______________________________________