[Python-Dev] Spurious Buildbot Reports

[Nick Coghlan]

Raymond Hettinger wrote:
> The bots are kicking-off so many false alarms that it is becoming
> difficult to tell whether a check-in genuinely broke a build.

It would also be nice if the checkins list only got spammed for actual 
compile or test failures. I'm not all that interested in getting an 
email just because a box got rebooted or lost its net connection for a 
while.

In terms of checking the buildbot status page itself, it would be a lot 
more convenient if the "current/last build" field on the buildbot slave 
details page was a hyperlink to that build's results page rather than a 
mere number as it is now.

> At the root of the problem is a number of tests in the test suite
> that randomly blow-up.  I now tend to automatically dismiss failures
> in test_logging and test_threading for example.

I haven't noticed that so much as the fact that a few of the buildbots 
have been pretty much permanently red for quite some time. Some examples:

alpha tru64:
   Checking the build results for this machine usually shows quite a few 
different errors, although the latest addition to the collection is a 
segfault in test_ctypes. Do we really support tru64 well enough to have 
it as a build slave?

ppc debian unstable:
   test_xmlrpc failing with connection reset by peer errors. I think 
I've seen that on other platforms as well, but this one seems to do it 
nearly constantly. (I don't know enough about the structure of that test 
to know if a GHOP student could figure out the problem without access to 
a machine where the test fails consistently - might be worth a try though)

MIPS debian:
   test_urllib2net FTP tests are failing, apparently complaining that 
SocketType expects two arguments rather than three. Probably a hard one 
to figure out without access to a machine where the test is actually 
failing (even my description of the problem involves some guesswork 
based on reading the code near where the failure occurs).

I think a few of the others (such as hppa Ubuntu) also have issues.

It's gotten to the point where I only pay any real attention to about 
half the trunk buildbots (looking at the status summary page, I would 
probably investigate further if sparc solaris, g4 osx 10, XP-3, XP-4 or 
x86 FreeBSD turned red after one of my checkins) and pretty much ignore 
the rest.

Would it be possible to distinguish the reliable buildbots from the 
problematic ones in the build master? If such a distinction can be made, 
it might then be possible to arrange for email to be sent to the checkin 
list only if one of the reliable buildbots fail.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org