[issue42109] Use hypothesis for testing the standard library, falling back to stubs

[Paul Ganssle]

New submission from Paul Ganssle <p.ganssle at gmail.com>:

Following up on this year's language summit, I would like to propose that we start integrating property tests into the standard library. Zac Hatfield-Dodds (along with myself and Carl Friedrich Bolz-Tereick) have put together this repository of tests that we run against the standard library as is: https://github.com/Zac-HD/stdlib-property-tests

Here is the blog post covering the proposal from the language summit: https://pyfound.blogspot.com/2020/05/property-based-testing-for-python.html

The biggest challenges here are logistical:

1. Pulling in third party dependencies is difficult to do reliably on CI, but checking in hypothesis's entire dependency tree is probably not feasible.
2. We don't necessarily want to require developers running their tests locally to have to set up a hypothesis environment just to run the tests.
3. Hypothesis tests are not (by default) deterministic, which some are concerned may lead to flakiness by themselves.


To allay these concerns, I propose that we implement a compatibility interface for hypothesis that uses the third party module when it's installed, but otherwise falls back to stubs. The way I see the stubs working is that `@given` *without* `@example`s would simply skip the test. If you specify `@given` and one or more `@example`s, the test falls back to a simple parameterized test when hypothesis is not available.

At least at first, we won't attempt to add a mandatory PR job that runs with hypothesis installed. Instead, I'd like to run either an optional job on PR or have one or more buildbots that runs the hypothesis tests.

I would also like to suggest a policy of including at least one example in each property test, so that on PR at least some of the inputs are tested.

----------
assignee: p-ganssle
components: Tests
messages: 379226
nosy: Zac Hatfield-Dodds, p-ganssle, terry.reedy
priority: normal
severity: normal
stage: patch review
status: open
title: Use hypothesis for testing the standard library, falling back to stubs
type: enhancement
versions: Python 3.10

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42109>
_______________________________________