[Python-Dev] Reporting unexpected import failures as test failures in regrtest.py

[Tim Peters]

[Nick Coghlan]
> What if we appended unexpected skips to the list of bad tests so that they get
> rerun in verbose mode and the return value becomes non-zero?
>
>      print count(len(surprise), "skip"), \
>                        "unexpected on", plat + ":"
>      printlist(surprise)
>      # Add the next line after the previous two in regrtest.py
>      bad.extend(surprise)
>
> (This happens after the count of failed tests has been printed, so we don't
> affect that output)

While "expected skips" is reliable on Windows, it's just a guess
everywhere else.  That's because the Windows build is unique in
supplying and compiling all the external packages Python ships with on
Windows.  On non-Windows boxes, which skips "are expected" also
depends on which external packages happen to be installed on the box,
and how Python was configured, and regrtest.py has no knowledge about
those.

For example, on the "sparc solaris10 gcc trunk" buildbot, 6 tests show
up as "unexpected skips", and there's just no way for a
non-platform-expert to guess which of those are and aren't pointing at
problems.  It's easy enough to guess that test_zipfile is skipped
because the box doesn't have an external zlib installed, but why is
test_ioctl skipped?  I have no idea (and neither does regrtest.py),
but it's presumably obvious to a Sun expert.

In any case, if "unexpected skips" got appended to `bad`, the non-zero
exit code would cause that buildbot slave to fail every time.  It's
not unique, either; e.g., the "x86 gentoo trunk" slave always skips
test_tcl, presumably because Tcl isn't installed on that box.

In short, there's simply no way to _guess_ which import errors are and
aren't "expected".  Improving that requires adding more knowledge to
the testing machinery.  The most important thing is to add knowledge
about which import errors are _never_ expected (perhaps by adding
knowledge about which imports _may_ legitimately fail), since we can
know that for sure about x-platform modules, and that's exactly the
problem that keeps burning us.  Confusing ImportError with TestSkipped
is an admirably lazy idea that unfortunately didn't turn out to work
very well.