[issue42109] Use hypothesis for testing the standard library, falling back to stubs

[Raymond Hettinger]

Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

Here's my two cents worth on the subject.  

* I use hypothesis during development, but don't have a need for in the the standard library.  By the time code lands there, we normally have a specific idea of what edge cases needs to be in the tests.

* For the most part, hypothesis has not turned up anything useful for the standard library.  Most of the reports that we've gotten reflected a misunderstanding by the person running hypothesis rather than an actual bug.  For example, in colorsys, an issue was reported about the color conversions not round-tripping, but that is an expected behavior for out of gamut conversions.  Also, the matrices used are dictated by standards and do not give exact inverses even for in gamut conversions.

* For numeric code, hypothesis is difficult to use and requires many restrictions on the bounds of variables and error tolerances.  For example, when I have students with a function to evaluate the quadratic formula and test it with hypothesis, they struggle mightily (overflowing the discriminant, having 2*a close to zero, deciding whether the roots are sufficiently close, etc.)

* The main area where hypothesis seems easy to use and gives some comfort is in simple roundtrips:  assert zlib.decompress(zlib.compress(s)) == s.  However, that is only a small fraction of our test cases.

* Speed is another issue.  During development, it doesn't matter much if Hypothesis takes a lot of time exercising one function.  But in the standard library tests already run slow enough to impact development.  If hypothesis were to run everytime we run a test suite, it would make the situation worse.

* There is also a learning curve issue.  We're adding more and more things that newcomer's have to learn before they can contribute (how to run blurb, how to use the argument clinic, etc).  Using Hypothesis is a learned skill and takes time.

TL;DR  I really like Hypothesis but think it doesn't belong in standard library tests.

----------
nosy: +rhettinger

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42109>
_______________________________________