[pytest-dev] Creating a complex PyTest program, wondering about the best way to handle collection errors, etc.

[Floris Bruynooghe]

Hi Nick,

On Wed 18 Apr 2018 at 13:19 -0500, Nicholas Williams wrote:

> Hello, all. This is my first post to the mailing list, FWIW.
>
> I've been a PyTest user for some years, but now I'm working on writing a
> rather complex PyTest plugin to facilitate testing our service oriented
> architecture at $work. The purpose of the plugin is to collect additional
> tests (for which we are using `pytest_collection_modifyitems`) from a

I think you'll fare better by integrating deeper into the collection
mechanism instead of using pytest_collection_modifyitems.  How are these
test written?  You probably should look at the pytest_collect_directory
and pytest_collect_file hooks to do this.  Unfortunately the
documentation on these things is very thin.  Finding existing code that
does this is probably your best bet.  The _pytest/python.py plugin is
the canonical example though is very complicated.  If you hunt other
plugins you may find other examples.

> specialized file format that PyTest wouldn't normally understand. I've got
> it working beautifully, but I would love to hammer out some finer details
> to really make it a clean-running plugin.
>
> With that in mind, I'm looking for some suggestions / best practices /
> advice here on how best to handle these situations:
>
> Collection Errors
> The author of one of these specialized tests, which we call "test plans,"
> could possibly do a few things wrong that cause more than just making their
> test fail. For example, they could get their file syntax incorrect, which
> results in a parsing error. At the moment, I catch the parsing error (we're
> using PyParsing to parse our test plans) and raise a different exception
> about the problem (and I've tried AssertionError), but everything I do
> results in a stack trace dumped with "INTERNALERROR>" and the immediate,
> abnormal termination of the PyTest process. This contrasts to, for an
> analog example, a syntax error in a regular Python test file, which instead
> results in a "ERROR collecting" output, while all other unaffected tests
> still run and the PyTest process terminates normally. What I'm looking for
> here is the "right way" to make my plugin report an "ERROR collecting"
> instead of causing an "INTERNALERROR>" when such a problem occurs
> collecting these specialized tests.

I think these problems would go away if you join the collection
mechanism "correctly".  Not that "correctly" is necessarily easy or
obvious... but the hints above might help you.

> Improving the Test Error Report

I think this was already answered.


Cheers,
Floris