[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

STINNER Victor added the comment:

> The exact behavior depends on the order of files in directories, on string hashes randomization, on address randomization, and on many other things out of our control.

For hash randomization, maybe we need to generate a PYTHONHASHSEED, as tox test runner does.

For the filesystem: right, it's not possible to get 100% reproductible tests, but IMHO it's worth it to make them more reliable.

> Couldn't reseeding the PRNG just add a false promise?

I'm trying to fix random failures on the Refleaks buildbots, not to promise anything :-) To be honest, at this point, I don't know if it would be enough since I'm unable to reproduce bugs...

> The success in making tests deterministic can also narrow down the coverage of the testing. Some branches that lead to failures can be never executed. Our target not just making tests always success, but catch and fix even pretty rare errors.

I know that it's though question, and that's why I opened this issue, to discuss it :-)

But I see more and more projects to get more reproductible softwares and tests:

* https://reproducible-builds.org/
* systemd big project to get more "stateless" computers, or said differently: to isolate better services
* containers which also want to isolate services, "stateless" containers
* etc.

Other test runners, like tox, also make efforts to get reproductible tests, like setting PYTHONHASHSEED.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue31227>
_______________________________________