[pytest-dev] Custom reporting for asserts without comparison operators?

[Shawn Brown]

Ah. It's good to see that this has been thought about before.

My motivation for asking this question was to perform my due diligence and
make sure I wasn't missing something before moving ahead. My immediate need
is handled by using assert_myfunc() to raise its own error internally--same
as Floris' example. Though, it's not ideal.

I know my examples have been vague as I've stripped the specifics of my
project to focus the question specifically on pytest's behavior and I
greatly appreciate everyone who is giving some thought to this.

As Ronny mentioned, I'm sure it's possible to address this without
user-facing AST manipulation. But I'm not familiar enough with the code
base to see where I can best hack on the representations. However, I do
have a working AST-based demonstration (below). This uses a fragile
monkey-patch that is just asking for trouble so please take this for the
experimental hack it is...


FILE "conftest.py":

    import ast
    import _pytest

    def my_ast_prerewrite_hook(ast_assert):
        """Modifies AST of certain asserts before pytest-rewriting."""
        # Demo AST-tree manipulation (actual implemenation
        # would need to be more careful than this).
        if (isinstance(ast_assert.test, ast.Call)
                and isinstance(ast_assert.test.func, ast.Name)
                and ast_assert.test.func.id == 'myfunc'):

            ast_assert.test.func = ast.Name('assert_myfunc', ast.Load())

        return ast_assert

    # UNDESIRABLE MONKEY PATCHING!!!
    class ModifiedRewriter(_pytest.assertion.rewrite.AssertionRewriter):
        def visit_Assert(self, assert_):
            assert_ = my_ast_prerewrite_hook(assert_)  # <- PRE-REWRITE HOOK
            return super(ModifiedRewriter, self).visit_Assert(assert_)

    def rewrite_asserts(mod, module_path=None, config=None):
        ModifiedRewriter(module_path, config).run(mod)

    _pytest.assertion.rewrite.rewrite_asserts = rewrite_asserts


FILE "test_ast_hook_approach.py":

    import pytest

    # Test helpers.
    def myfunc(x):
        return x == 42

    def assert_myfunc(x):
        __tracebackhide__ = True
        if not myfunc(x):
            msg = 'custom report\nmulti-line output\nmyfunc({0}) failed'
            raise AssertionError(msg.format(x))
        return True

    # Test cases.
    def test_1passing():
        assert myfunc(42)

    def test_2passing():
        assert myfunc(41) is False

    def test_3passing():
        with pytest.raises(AssertionError) as excinfo:
            assert myfunc(41)
        assert 'custom report' in str(excinfo.value)

    def test_4failing():
        assert myfunc(41)


Running the above test gives 3 passing cases and 1 failing case (which uses
the custom report). Also, test_2passing() checks for "is False" instead of
just "== False" which I think would be wonderful to support as it removes
all caveats for the user (so users get a real False when they expect False,
instead of a Falsey alternative). Also, if I were going to use AST
manipulation like this, I would probably reference assert_myfunc() by
attaching it as a private attribute to myfunc() itself -- and then
reference it with ast.Attribute() node instead of an ast.Name(). But again,
solving this without AST manipulation could be better in many ways.

--Shawn


On Mon, Mar 19, 2018 at 1:59 PM, Ronny Pfannschmidt <
ich at ronnypfannschmidt.de> wrote:

> hi everyone,
>
> this is just about single value assertion helpers
>
> i logged an feature request about that a few year back
> see https://github.com/pytest-dev/pytest/issues/95 -
>
> so basically this use-case was known since 2011 ^^ and doesn't require
> ast rewriting lice macros,
> just proper engineering of the representation and handling of single
> values in the assertion rewriter.
>
> -- Ronny
>
>
> Am 19.03.2018 um 15:13 schrieb holger krekel:
> > On Mon, Mar 19, 2018 at 15:03 +0100, Floris Bruynooghe wrote:
> >> On Sun, Mar 18 2018, Shawn Brown wrote:
> >>> Unfortunately, this does not solve my usecase. I'm trying to handle
> cases
> >>> where the following statement would pass:
> >>>
> >>>     assert myfunc(failing_input) == False
> >>>
> >>> But where this next statement would fail using my custom report:
> >>>
> >>>     assert myfunc(failing_input)
> >>>
> >>> Calling myfunc() needs to return True or False (or at least Truthy or
> >>> Falsy)--this is locked-in behavior.
> >> I'm not sure if this is compatible with Python's semantics really.  If I
> >> understand correctly you're asking for a full-on macro implementation on
> >> Python or something.  Which in theory you could do with an AST
> >> NodeVisitor, but really Python isn't made for this -- sounds like you'd
> >> enjoy lisp! ;-)
> >>
> >> The best thing I can suggest is to make use of the::
> >>
> >>    assert myfunc(failing_input), repr(myfunc(failing_input()))
> > i wonder if one could try to rewrite the ast for "assert myfunc(x)" to
> > "assert __pytest_funcrepr_helper(myfunc(x), 'myfunc(x)')" with
> something like:
> >
> >     class __pytest_funcrepr_helper:
> >         def __init__(self, val, source):
> >             self.val = val
> >             self.source = source
> >         def __bool__(self):
> >             return bool(self.val)
> >         def __repr__(self):
> >             return "{!r} returned non-true {!r}".format(self.source,
> self.val)
> >
> > but maybe i am not grasping all details involved. It's been a while since
> > i looked into ast-rewriting ...
> >
> > holger
> >
> >
> >> functionality to also get a custom error message.  Here your myfunc()
> >> whould have to return some object which both implements __bool__ as well
> >> as __repr__ I guess.
> >>
> >> Maybe there's a feature request in here for something like this::
> >>
> >>    class Foo:
> >>        def __bool__(self):
> >>            return False
> >>
> >>        def __repr__(self):
> >>            return 'multiline\nstring'
> >>
> >>    assert Foo()
> >>
> >> To actually show the repr in the error message, which it currently
> >> doesn't.  I'd like to know what other people think of such a feature
> >> though, and haven't thought through all the implications yet.  But I'm
> >> curious, would something like that solve your case?
> >>
> >> Cheers,
> >> Floris
> >> _______________________________________________
> >> pytest-dev mailing list
> >> pytest-dev at python.org
> >> https://mail.python.org/mailman/listinfo/pytest-dev
> > _______________________________________________
> > pytest-dev mailing list
> > pytest-dev at python.org
> > https://mail.python.org/mailman/listinfo/pytest-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20180321/9f261c00/attachment-0001.html>