[pytest-dev] Improving pytest internal errors and messages

[Bruno Oliveira]

Hi everyone,

Often some errors in pytest show a long traceback with the pytest
internals, which is noisy and distracting.

For example, if a user makes a typo when declaring the scope of a fixture,
we get an error like this:

```
============================================= ERRORS
==============================================
_____________________________ ERROR collecting test-scope-mismatch.py
_____________________________
src\_pytest\fixtures.py:699: in scope2index
    return scopes.index(scope)
E   ValueError: 'modu' is not in list

During handling of the above exception, another exception occurred:
src\_pytest\runner.py:201: in __init__
    self.result = func()
src\_pytest\runner.py:265: in <lambda>
    call = CallInfo(lambda: list(collector.collect()), "collect")
src\_pytest\python.py:475: in collect
    self.session._fixturemanager.parsefactories(self)
src\_pytest\fixtures.py:1321: in parsefactories
    ids=marker.ids,
src\_pytest\fixtures.py:837: in __init__
    scope or "function", descr="fixture {}".format(func.__name__),
where=baseid
src\_pytest\fixtures.py:703: in scope2index
    descr, "from {} ".format(where) if where else "", scope
E   ValueError: fixture fix from test-scope-mismatch.py has an unsupported
scope value 'modu'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================== 1 error in 0.16 seconds
=====================================
```

I've opened a WIP PR (https://github.com/pytest-dev/pytest/pull/4077) to
deal with this which uses a specific exception type to raise errors in
pytest that are "user errors", suppressing the full traceback unless `-vvv`
is used.

```
============================================= ERRORS
==============================================
_____________________________ ERROR collecting test-scope-mismatch.py
_____________________________
fixture fix from test-scope-mismatch.py has an unsupported scope value
'modu'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================== 1 error in 0.07 seconds
=====================================
```

This new exception type (I've used UsageError on my PR just for proof of
concept) should be part of the public API so plugins can also use it.

I'm writing to the ML because it would be beneficial to get feedback from a
larger audience about what the *name* of the exception should be.

UserError? InternalError?

Feedback on the overall approach (exception usage, verbosity handling, etc)
is also very welcome!

(On the PR there is a list of issues that could use the new mechanism to
provide better messages)

Cheers,
Bruno.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20181004/689cfb88/attachment.html>