[pypy-dev] xfail versus skips

[Maciej Fijalkowski]

On Thu, May 27, 2010 at 8:34 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Hi all,
>
> On 05/25/2010 10:22 PM, holger krekel wrote:
>> i just released py.test-1.3.1 [1] which is inlined as svn/pypy/trunk/py
>> and the pypy/test_all.py script is an alias for "py.test".  This release
>> particularly refines "expected-to-fail" aka "xfail" semantics:
>>
>>      # abort setup or test function, reporting as "expected to fail", or 'x'
>>      py.test.xfail() or py.test.xfail(reason)
>>
>> see http://codespeak.net/py/dist/test/plugin/skipping.html
>> for more details.  Marking tests as 'xfail' is also good for
>> tests that *sometimes* fail.
>>
>> I just did a grep of "py.test.skip" in pypy/trunk/pypy and
>> there are 468 occurences [2].  Many of these skips seem to be because
>> of implementation issues rather than platform/dependency mismatches
>> and should thus rather use py.test.xfail.  Being Skips kind of hides
>> those issues between the rightful skips.  The xfail/skip distinction is
>> something that is happening in other parts of the Python world as well
>> and i hope you find it useful as well.
>
> I think another thing is that many of the tests that are now skipped
> should really be deleted, because they are completely outdated or
> because it just does not make sense to support them.
>

... but the sheer number of skips means that we don't even want to
look at them (which was the xfail part trying to mitigate).

Cheers,
fijal