[pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc.

[Ringo De Smet]

Nicholas,

Regarding the stack trace filtering, I have been bitten by this too in my
custom plugin. You can read the solution here in the mail archives:

https://mail.python.org/pipermail/pytest-dev/2018-March/004399.html

If it is not clear enough, just reply and I might be able to help you
forward.

Ringo


On Wed, Apr 18, 2018 at 8:19 PM, Nicholas Williams <
nicholas+pytest at nicholaswilliams.net> wrote:

> Hello, all. This is my first post to the mailing list, FWIW.
>
> I've been a PyTest user for some years, but now I'm working on writing a
> rather complex PyTest plugin to facilitate testing our service oriented
> architecture at $work. The purpose of the plugin is to collect additional
> tests (for which we are using `pytest_collection_modifyitems`) from a
> specialized file format that PyTest wouldn't normally understand. I've got
> it working beautifully, but I would love to hammer out some finer details
> to really make it a clean-running plugin.
>
> With that in mind, I'm looking for some suggestions / best practices /
> advice here on how best to handle these situations:
>
> Collection Errors
> The author of one of these specialized tests, which we call "test plans,"
> could possibly do a few things wrong that cause more than just making their
> test fail. For example, they could get their file syntax incorrect, which
> results in a parsing error. At the moment, I catch the parsing error (we're
> using PyParsing to parse our test plans) and raise a different exception
> about the problem (and I've tried AssertionError), but everything I do
> results in a stack trace dumped with "INTERNALERROR>" and the immediate,
> abnormal termination of the PyTest process. This contrasts to, for an
> analog example, a syntax error in a regular Python test file, which instead
> results in a "ERROR collecting" output, while all other unaffected tests
> still run and the PyTest process terminates normally. What I'm looking for
> here is the "right way" to make my plugin report an "ERROR collecting"
> instead of causing an "INTERNALERROR>" when such a problem occurs
> collecting these specialized tests.
>
> Improving the Test Error Report
> When tests have errors or failures, PyTest prints out a report of those
> errors, and with each one includes an exhaustive stack trace. For our
> specialized tests, with some clever manipulation of the test data added to
> `items` in `pytest_collection_modifyitems`, I've managed to at least get
> it to display a meaningful name for the test that failed so that it's at
> least _possible_ for the developer to figure out which test, in which test
> file, failed. However, almost the entire stack trace is meaningless and
> full of plugin code instead of containing information useful to debugging
> the test. Additionally, I would _love_ to be able to add the actual test
> file and line number for the failure to the report. Would this be possible?
> Or, at least, would it be possible to filter the plugin code out of the
> stack trace so that it doesn't distract the developer from simply viewing
> the assertion error details? What approach should I take to achieve this?
>
> Thanks in advance for any help you can provide,
>
> Nick
>
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
>
>


-- 
*Ringo De Smet*
ringo.de.smet at ontoforce.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20180419/26a8dd90/attachment-0001.html>