[Pytest-commit] Issue #449: better indicate that there were xpassed test results in a test run (hpk42/pytest)

[Jurko Gospodnetić]

New issue 449: better indicate that there were xpassed test results in a test run
https://bitbucket.org/hpk42/pytest/issue/449/better-indicate-that-there-were-xpassed

Jurko Gospodnetić:

This is a usability related enhancement suggestion.

When running tests what I really want to know first is:

* Did anything unexpected happen?

which includes both:

* Whether there were any test failures?
* Whether any tests marked ```xfail``` suddenly started passing?

The first I can get easily at first glance - either by running the tests with the ```-x``` option or by checking the final summary line color (green/red). The latter is a problem though because in order to find whether there were any ```xpassed``` test I have to concentrate a lot harder and either:

* Scan the test summary for ```xpassed``` test result indicators (capital letter ```X```) which can be difficult to discern from expected ```xfailed``` (lower letter ```x```) results.
* Read the final colored summary line to see if any ```xpassed``` test results occurred.
* Run the tests with the '-r X' option and read whether the summary displays any ```xpassed``` test results.

One of my projects has a lot of tests marked ```xfail``` and it started to bug me that I often waste a lot of time and interrupt my flow & concentration by having to check the test results in detail just to see if there were any ```xpassed``` test results.

My suggestion would be to:

  * Use a different color (e.g. blue?) if you would otherwise color it green by at least one ```xpassed``` test result was encountered.
  * Use an exit code other that a simple 0=success in such cases.
  * Possibly color individual ```failed``` (```F```) & ```xpassed``` (```X```) test result indicators red or red/blue.

  You might not want to use the new ```xpassed``` result related coloring when running with disabled assertions (```-O```) and you displayed a warning about this possibly causing failing tests to be marked as passed, e.g. when running using an older Python interpreter version.

  The whole enhancement could possibly be made configurable as well.

  Hope this helps.

  Best regards,
    Jurko Gospodnetić