[Python-Dev] Cloud build slaves (was Re: Possible language summit topic: buildbots)

[exarkun at twistedmatrix.com]

On 04:31 pm, ctb at msu.edu wrote:
>On Fri, Oct 30, 2009 at 04:21:06PM +0000, Antoine Pitrou wrote:
>>
>>Hello,
>>
>>Sorry for the little redundancy, I would like to underline Jean-Paul's
>>suggestion here:
>>
>>Le Sun, 25 Oct 2009 14:05:12 +0000, exarkun a ??crit??:
>> > I think that money can help in two ways in this case.
>> >
>> > First, there are now a multitude of cloud hosting providers which 
>>will
>> > operate a slave machine for you.  BuildBot has even begun to support
>> > this deployment use-case by allowing you to start up and shut down 
>>vms
>> > on demand to save on costs.  Amazon's EC2 service is supported out 
>>of
>> > the box in the latest release.
>>
>>I'm not a PSF member, but it seems to me that the PSF could ask Amazon
>>(or any other virtual machine business anyway) to donate a small 
>>number
>>of permanent EC2 instances in order to run buildslaves on.
>
>[ ... ]
>
>I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); 
>Linux
>ia64; Linux x86; and Mac OS X.

Okay, let's move on this.  Martin has, I believe, said that potential 
slave operators only need to contact him to get credentials for new 
slaves.  Can you make sure to follow up with him to get slaves running 
on these machines?  Or would you rather give out access to someone else 
and have them do the build slave setup?
>Others have made similar offers.

I'll similarly encourage them to take action, then.  Do you happen to 
remember who?
>The
>architectures supported by the cloud services don't really add anything
>(and generally don't have Mac OS X support, AFAIK).

That's not entirely accurate.  Currently, CPython has slaves on these 
platforms:

  - x86
    - FreeBSD
    - Windows XP
    - Gentoo Linux
    - OS X
  - ia64
    - Ubuntu Linux
  - Alpha
    - Debian Linux

So, assuming we don't want to introduce any new OS, Amazon could fill in 
the following holes:

  - x86
    - Ubuntu Linux
  - ia64
    - FreeBSD
    - Windows XP
    - Gentoo Linux

So very modestly, that's 4 currently missing slaves which Amazon's cloud 
service *does* add.  It's easy to imagine further additions it could 
make as well.
>What we really need (IMO) is someone to dig into the tests to figure 
>out which
>tests fail randomly and why, and to fix them on specific architectures 
>that
>most of us don't personally use.  This is hard work that is neither 
>glamorous
>nor popular.

Sure.  That's certainly necessary.  I don't think anyone is suggesting 
that it's not.  Fortunately, adding more build slaves is not mutually 
exclusive with a developer fixing bugs in CPython.
>I think the idea of paying a dedicated developer to make the 
>CPython+buildbot
>tests reliable is better, although I would still be -0 on it (I don't 
>think
>the PSF should be paying for this kind of thing at all).

I hope everyone is on board with the idea of fixing bugs in CPython, 
either in the actual implementation of features or in the tests for 
those features.  That being the case, the discussion of whether or not 
the PSF should try to fund such a task is perhaps best discussed on the 
PSF members list.

Jean-Paul