[issue42109] Use hypothesis for testing the standard library, falling back to stubs

Wed May 26 01:36:26 EDT 2021

Terry J. Reedy <tjreedy at udel.edu> added the comment:

In order to understand what Paul is concretely proposing, I read and partly reviewed his PR.  I thought about maintainability.

Part A adds test.support.hypothesis_helper, similar in purpose to other specialized xyz_helper modules.  Hypothesis_helper imports either hypothesis itself or a replacement 'as hypothesis'.  The replacement is currently called '_hypothesis_stub', but its name is irrelevant as it is normally not seen by the user.

Part B adds a second zoneinfo test module, one that uses hypothesis. Besides further testing zoneinfo, it serves to test hypothesis_helper and partly justify its addition.  It starts with 'from hypothesis_helper import hypothesis.  The new test module has tests such as

    @hypothesis.given(
        dt=hypothesis.strategies.one_of(
            hypothesis.strategies.datetimes(), hypothesis.strategies.times()
        )
    )
    @hypothesis.example(dt=datetime.datetime.min)
    @hypothesis.example(dt=datetime.datetime.max)
    @hypothesis.example(dt=datetime.datetime(1970, 1, 1))
    @hypothesis.example(dt=datetime.datetime(2039, 1, 1))
    @hypothesis.example(dt=datetime.time(0))
    @hypothesis.example(dt=datetime.time(12, 0))
    @hypothesis.example(dt=datetime.time(23, 59, 59, 999999))
    def test_utc(self, dt):
        zi = self.klass("UTC")
        dt_zi = dt.replace(tzinfo=zi)

        self.assertEqual(dt_zi.utcoffset(), ZERO)
        self.assertEqual(dt_zi.dst(), ZERO)
        self.assertEqual(dt_zi.tzname(), "UTC")

@given always (Paul?) runs the examples as subtests.  When the replacement is imported, no randomized examples are added.

If some year an example were to fail, could a hypothesis-ignorant zoneinfo-knowing person deal with the failure?  I believe so, knowing that the above is equivalent to a test with "for dt in dt_list:\n  with self.subTest(...):\n    <body of function>"

Why not require such a rewriting?  Some reasons against: Rewriting by hand can lead to errors.  test_utc would have to be split into test_utc_ran(dom) and test_utc_pre(set).  Code would have to be duplicated unless factored into a third function.  For and with together add two indents, which sometimes squeezes assert onto more lines.  I believe that when hypothesis is present, there are advantages to including preset examples with given-generated examples.

Paul would like the PR to be a 'camel's nose' in the CPython tent.  This cuts both ways.  I think this PR should be judged on its own merits.  Opening possibilities can be a merit as long as not seen as pre-approval.  Our CI testing and buildbots are already configured to blame and annoy the wrong people for random failures.  I don't want more unless failure notices are sent to someone responsible for the failing code.

----------
versions: +Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42109>
_______________________________________