[Python-Dev] Classifying language vs. impl-detail tests

[Armin Rigo]

Hi all,

Here is a first step towards classifying the Python test suite into
"real language tests" and "implementation details":

    http://bugs.python.org/issue4242

If the general approach seems acceptable to people, I would be willing
to port more of PyPy's test suite patches.  The net result would be to
move the test suite towards being more directly useful for alternate
Python implementations.  (Right now, all of them have some custom tests
plus their own similarly patched copy of the stdlib test suite.)

Note that the patch above is against 2.7 trunk; the 2.x line is what all
non-CPython implementations currently target.  The general idea (and to
a large extend the patches) can easily be forward-ported to 3.x when it
makes sense.

Also, the actual decision of what is a "real" or a "detail" test is of
course up to discussion.  The patch above does the classification for
test_descr.  It was obtained by running it with PyPy, and for each
failure either fixing PyPy, or argumenting (by adding a comment) the
reason for which it is a detail -- usually referring to "well-known
agreement" expressed on python-dev about the matter.  (Of course,
"detail" tests still run normally on top of CPython.)


A bientot,

Armin.