[Python-Dev] Possible language summit topic: buildbots

[exarkun at twistedmatrix.com]

On 09:47 am, martin at v.loewis.de wrote:
>Mark Dickinson wrote:
>>Would it be worth spending some time discussing the buildbot situation
>>at the PyCon 2010 language summit?  In the past, I've found the
>>buildbots to be an incredibly valuable resource;  especially when
>>working with aspects of Python or C that tend to vary significantly
>>from platform to platform (for me, this usually means floating-point,
>>and platform math libraries, but there are surely many other things it
>>applies to).  But more recently there seem to have been some
>>difficulties keeping a reasonable number of buildbots up and running.
>>A secondary problem is that it can be awkward to debug some of the
>>more obscure test failures on buildbots without having direct access
>>to the machine.  From conversations on IRC, I don't think I'm alone in
>>wanting to find ways to make the buildbots more useful.
>
>These are actually two issues:
>a) where do we get buildbot hardware and operators?
>b) how can we reasonably debug problems occurring on buildbots
>
>For a), I think we can solve this only by redundancy, i.e. create
>more build slaves, hoping that a sufficient number would be up at
>any point in time.
>
>So: what specific kinds of buildbots do you think are currently 
>lacking?
>A call for volunteers will likely be answered quickly.
>>So the question is: how best to invest time and possibly money to
>>improve the buildbot situation (and as a result, I hope, improve the
>>quality of Python)?
>
>I don't think money will really help (I'm skeptical in general that
>money helps in open source projects). As for time: "buildbot scales",
>meaning that the buildbot slave admins will all share the load, being
>responsible only for their own slaves.

I think that money can help in two ways in this case.

First, there are now a multitude of cloud hosting providers which will 
operate a slave machine for you.  BuildBot has even begun to support 
this deployment use-case by allowing you to start up and shut down vms 
on demand to save on costs.  Amazon's EC2 service is supported out of 
the box in the latest release.

Second, there are a number of active BuildBot developers.  One of them 
has even recently taken a contract from Mozilla to implement some non- 
trivial BuildBot enhancements.  I think it very likely that he would 
consider taking such a contract from the PSF for whatever enhancements 
would help out the CPython buildbot.
>On the master side: would you be interested in tracking slave admins?
>>What could be done to make maintenance of build
>>slaves easier?
>
>This is something that only the slave admins can answer. I don't think
>it's difficult - it's just that people are really unlikely to 
>contribute
>to the same thing over a period of five years at a steady rate. So we
>need to make sure to find replacements when people drop out.

This is a good argument for VMs.  It's certainly *possible* to chase an 
ever changing set of platforms, but it strikes me as something of a 
waste of time.
>The source of the problem is that such a system can degrade without
>anybody taking action. If the web server's hard disk breaks down,
>people panic and look for a solution quickly. If the source control
>is down, somebody *will* "volunteer" to fix it. If the automated
>build system produces results less useful, people will worry, but
>not take action.

To me, that raises the question of why people aren't more concerned with 
the status of the build system.  Shouldn't developers care if the code 
they're writing works or not?

Jean-Paul