[Python-Dev] Community buildbots and Python release quality metrics

[Steve Holden]

Georg Brandl wrote:
> glyph at divmod.com schrieb:
> 
>> Another way to phrase this question is, "whose responsibility is it to 
>> make Python 2.5 programs run on Python 2.6"?  Or, "what happens when 
>> the core team finds out that a change they have made has broken some 
>> python software 'in the wild'"?
>>
>> Here are a couple of ways to answer these questions:
>>
>>   1) It's the python core team's responsibility.  There should be 
>> Python buildbots for lots of different projects and they should all be 
>> green before the code can be considered [one of: allowed in trunk, 
>> alpha, beta, RC].
>>   2) It's the python core team's responsibility as long as the major 
>> projects are using public APIs.  The code should not be considered 
>> [alpha, beta, RC] if there are any known breakages in public APIs.
>  >
>>   3) It's only the python core team's responsibility to notify 
>> projects of incompatible changes of which they are aware, which may 
>> occur in public or private APIs.  Major projects with breakages will 
>> be given a grace period before a [beta, RC, final] to release a 
>> forward-compatible version.
>>   4) The status quo: every new Python release can (and will, at least 
>> speaking in terms of 2.6) break lots of popular software, with no 
>> warning aside from the beta period.  There are no guarantees.
> 
> Python's backwards compatibility has never extended to non-public APIs.
> That rules out 1). 2) would be nice, but to be honest, 2) and 3) are
> unrealistic given the number of Python core developers and the tasks
> already at hand. 4) is not acceptable either.
> 
> However, you seem to be overlooking that there's a 3.5) in there: it's
> the Python core team's responsibility as long as the major projects are
> using public APIs; but to reduce the workload every project should watch
> its own buildbots and notify the core team using the issue tracker in
> order to get the issue resolved.
> 
> At no time will a policy "the community buildbots must be green" be
> useful: the tests that run on these buildbots are not under our control,
> so if the tests test things we deem non-public we can't do anything
> about it. (And we may have a hard time convincing project authors to
> change the tests if we promised to make them run anyway.)
> 
If, however, the community buildbot goes red because of a regression or 
backwards incompatibility the developers should, when notified, at least 
undertake to verify that the breakage is intentional.

Unfortunately having a buildbot is like installing a firewall. Some 
people assume that the very presence of the buildbot is a guard against 
version incompatibilities, whereas in truth there is always the messy 
business of communication required to ensure the information gets to 
where it is useful.

The bottom line, of course, is that if project X can't be bothered to 
tell the core development team when changes break their build it's 
nobody's fault but their own. If the report it and the core developers 
ignore the reports, that's something that should be rectified.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/