[pytest-dev] Using pytest_assertrepr_compare() for marked tests only?

[Shawn Brown]

Floris and Florian,

Thanks for the feedback. As I mentioned, this is something I was playing
with as an experiment. My specific use case would not be satisfied with
this change but I thought I'd ask if this were possible since it was an
idea I stumbled across while investigating my usecase. I will, however,
detail my usecase just so you know what I'm working on.

I've been developing a package that repurposes software testing practices
for data validation (https://datatest.rtfd.io/).

A simple example using the datatest's validate() function:

    from datatest import validate

    def test_somedata():
        data = {'A', 'B', 'C', 'D'}
        requirement = {'A', 'B'}
        validate(data, requirement)

The test above will fail with the following exception:

    ValidationError: does not satisfy set membership (2 differences): [
        Extra('C'),
        Extra('D'),
    ]

The ValidationError is a subclass of AssertionError. It contains one item
for each element in `data` that fails to satisfy the `requirement`.

I was playing around with the idea of a marker to implement something like
the following (see below) but still generate the ValidationError output via
pytest's plugin system:

    import pytest

    @pytest.mark.validate
    def test_somedata():
        data = {'A', 'B', 'C', 'D'}
        requirement = {'A', 'B'}
        assert data == requirement

The idea is to use an `assert` statement but still get the ValidationError
output (although there are many cases where using "==" is semantically
wrong so the validate() function wouldn't go away in all cases). While I
could emulate this type of output with pytest_assertrepr_compare(), I need
to do more than emulate the error.

Here's where things get more involved: The datatest package also implements
a system of acceptances that operate on the items contained within the
ValidationError. If a user decides that the "Extra" items are acceptable,
they can use the accepted() context manager:

    from datatest import validate, accepted, Extra

    def test_somedata():
        data = {'A', 'B', 'C', 'D'}
        requirement = {'A', 'B'}
        with accepted(Extra):
            validate(data, requirement)

In this case, the accepted() context manager accepts all of the items in
the error (which suppresses the error entirely). But if it contained other
items that were not accepted, then an error would be re-raised with the
remainder of the items.

Below, I've added comments showing why the assertrepr comparison can't work
with the context manager:

    import pytest
    from datatest import validate, accepted, Extra

    @pytest.mark.validate
    def test_somedata():
        data = {'A', 'B', 'C', 'D'}
        requirement = {'A', 'B'}
        with accepted(Extra):  # <- Never receives an inspectible error!
            assert data == requirement  # <- Raises an assertion that can
not be inspected or changed!

To implement my dream "validate marker", I would need to manipulate the
assertion inside the scope of the context manager--which is what the
validate() function does. But doing this is far from what
pytest_assertrepr_compare() has access to.

One solution I could see is an entirely new hook:

    def pytest_do_assertion(item, op, left, right=None):
        """ perform comparison

        Returns an AssertionError or None """

But from what I can see, implementing a hook like `pytest_do_assertion`
could require unreasonably invasive changes to pytest. That said, I would
be very interested if something like `pytest_do_assertion()` were possible
to add but I understand if it's not. Also, the hook name is just a
throwaway idea, it's probably not appropriate.

-Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20190516/1389fd6a/attachment.html>