[Python-Dev] Possible language summit topic: buildbots

[David Bolen]

"Martin v. Löwis" <martin at v.loewis.de> writes:

> The remaining issue is the popups; if a process still has a popup,
> you can't even terminate it properly. There are two kinds of popups:
> system-generated ones, and CRT-generated ones. For the CRT ones, we
> once had a way to turn them off, but I'm not sure whether that mechanism
> might have been removed. For the system messages, there is a way to
> turn them off in the parent process. David Bolen (IIRC) had developed
> a patch, but I think this patch only runs on his system(s).

Yes, process-stopping dialogs have probably been the single most
annoying issue over time running a Windows build slave - certainly
from my perspective in terms of maintenance and detection they have
taken up the largest amount of time.

I believe the CRT disabling is still active in the 3.x branches (the
"-n" flag to regrtest in the buildbot's test.bat), after you restored
it this past March (it had regressed during an earlier py3k branch set
of patches and caused a bunch of problems for a bit) but not in the
2.x branches or trunk.  So there's still a bit of exposure there and
I'd certainly be in favor of porting the regrtest -n support over to
all current development branches.

I think the other issue most likely to cause a perceived "downtime"
with the Windows build slave that I've had a handful of cases over the
past two years where the build slave appears to be operating properly,
but the master seems to just queue up jobs as if it were down.  The
slave still shows an established TCP link to the master, so I
generally only catch this when I happen to peek at the status web
page, or catch a remark here on python-dev, so that can reduce
availability.

My build slave (based on 0.7.5 I think) runs with local patches to:

1. Protect again Win32 pop-up error boxes in child processes.
2. A fix for a recursive chaining of Twisted Deferreds during uploads which
   could break for large file transfers.  This only came up when my build
   slave was generating daily MSI builds and uploading them to the master.
3. Handle clock jumps.  It's a general flaw in the presence of system clock
   adjustments, but I only encountered it while in my FreeBSD build slave
   under VMWare with a Linux host.

(2) and (3) are useful, but not likely to be an issue with most build
slaves in normal operation.  (2) probably isn't needed on my end any
more now that the daily MSI builds aren't run, and it's possible that
it got corrected in later buildbot updates, since I did report it on
the development list at the time.

(1) is a pretty trivial patch, but added a dependency on pywin32, so
passing it back up to the buildbot maintainers (back in 2007) stalled
while determining if that was ok, and I don't I ever closed the loop.
I did later make it fall back to ctypes if pywin32 was missing, but
then I think buildbot was using a minimum of Python 2.3 at the time,
so even ctypes was a new dependency.  Anyway, it became less crucial
when Python's regrtest started executing similar code, though the
buildbot patch covers anything run under it and not just the python
process.

I'd of course be happy to pass along the patch to anyone interested.

I believe that Thomas Heller had run his Windows buildbot with some
similar local code, but implemented with a modified buildbot script
for building Python, rather than tweaking buildbot itself.

Of course, the patch only protects against system pop-ups - it can't
control the CRT assertion dialogs when Python is built in debug mode,
which is why I've argued in the past that the test process for Python
should ensure those are disabled.  The CRT errors themselves are still
important, but can be redirected to stderr rather than a blocking GUI
dialog.

-- David