[pypy-dev] xfail versus skips

[Carl Friedrich Bolz]

Hi all,

On 05/25/2010 10:22 PM, holger krekel wrote:
> i just released py.test-1.3.1 [1] which is inlined as svn/pypy/trunk/py
> and the pypy/test_all.py script is an alias for "py.test".  This release
> particularly refines "expected-to-fail" aka "xfail" semantics:
>
>      # abort setup or test function, reporting as "expected to fail", or 'x'
>      py.test.xfail() or py.test.xfail(reason)
>
> see http://codespeak.net/py/dist/test/plugin/skipping.html
> for more details.  Marking tests as 'xfail' is also good for
> tests that *sometimes* fail.
>
> I just did a grep of "py.test.skip" in pypy/trunk/pypy and
> there are 468 occurences [2].  Many of these skips seem to be because
> of implementation issues rather than platform/dependency mismatches
> and should thus rather use py.test.xfail.  Being Skips kind of hides
> those issues between the rightful skips.  The xfail/skip distinction is
> something that is happening in other parts of the Python world as well
> and i hope you find it useful as well.

I think another thing is that many of the tests that are now skipped 
should really be deleted, because they are completely outdated or 
because it just does not make sense to support them.

Cheers,

Carl Friedrich