[Python-Dev] Community buildbots and Python release quality metrics

[Nick Coghlan]

glyph at divmod.com wrote:
> Today on planetpython.org, Doug Hellman announced the June issue of 
> Python magazine.  The cover story this month is about Pybots, "the 
> fantastic automation system that has been put in place to make sure new 
> releases of Python software are as robust and stable as possible".
> 
> Last week, there was a "beta" release of Python which, according to the 
> community buildbots, cannot run any existing python software.  Normally 
> I'd be complaining here about Twisted, but in fact Twisted is doing 
> relatively well right now; only 80 failing tests.  Django apparently 
> cannot even be imported.

beta 1 has some trouble running *our* test suite - I'd be fairly 
surprised if the community buildbots were in significantly better shape.

> The community buildbots have been in a broken state for months now[1].

Continuously running community buildbots on the maintenance trees makes 
sense, since those trees should always be in a releasable state. For the 
trunk, they're really only interesting when the Python core buildbots 
are reporting all green, but some of the community buildbots are 
reporting red.

One of the problems is what the term "beta" means to different groups - 
for us, this first beta was really about saying "zero new features from 
here on, focus on making what we have now work properly". The relatively 
late landing of a couple of major PEPs (371, 3108) also didn't do any 
favours for trunk stability.

If the community buildbots aren't largely green by the time beta 2 comes 
out, that's when I'll agree we have a problem - they should definitely 
be green by the time first release candidate comes out.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org