[Python-Dev] Community buildbots and Python release quality metrics

[glyph at divmod.com]

On 03:42 pm, ncoghlan at gmail.com wrote:
>glyph at divmod.com wrote:

>beta 1 has some trouble running *our* test suite - I'd be fairly 
>surprised if the community buildbots were in significantly better 
>shape.

That's another problem, yes :)
>>The community buildbots have been in a broken state for months now[1].

>If the community buildbots aren't largely green by the time beta 2 
>comes out, that's when I'll agree we have a problem - they should 
>definitely be green by the time first release candidate comes out.

I have mixed feelings about this: on the one hand, if this were a matter 
of public record, I would have known that this was too early to start 
complaining, and we could have saved everyone a lot of time reading 
these messages.  That would be good; I would prefer to just let everyone 
work without bothering about process.

On the other hand, it's much easier to deal with test failures as they 
arise than in one giant chunk of work at the end of the development 
process.  I spoke to a few core developers at PyCon who thought that the 
buildbots should always be green and any change that broke them should 
be reverted.  They may remain nameless unless they wish to raise their 
hands :-) but that's definitely how I feel about it.
>Continuously running community buildbots on the maintenance trees makes 
>sense, since those trees should always be in a releasable state. For 
>the trunk, they're really only interesting when the Python core 
>buildbots are reporting all green, but some of the community buildbots 
>are reporting red.

(... which should be all the time ...)
>One of the problems is what the term "beta" means to different groups - 
>for us, this first beta was really about saying "zero new features from 
>here on, focus on making what we have now work properly". The 
>relatively late landing of a couple of major PEPs (371, 3108) also 
>didn't do any favours for trunk stability.

A big part of why I wrote this message is that I wanted a clear 
understanding of the relationship between the definition of  "alpha", 
"beta" and "RC" and the state of various buildbots.  If that 
relationship exists already, just linking to it from 
http://python.org/download/releases/2.6/ would be good.  By the way, 
that page still says these are "alpha" releases.