[Python-Dev] Community buildbots and Python release quality metrics

[Georg Brandl]

glyph at divmod.com schrieb:
> On 07:44 pm, g.brandl at gmx.net wrote:
>>At no time will a policy "the community buildbots must be green" be
>>useful: the tests that run on these buildbots are not under our 
>>control,
>>so if the tests test things we deem non-public we can't do anything
>>about it. (And we may have a hard time convincing project authors to
>>change the tests if we promised to make them run anyway.)
> 
> That's not what I'm suggesting.

Sure, but I wanted to say that nevertheless :)

> If there is a legitimate disagreement between Python developers and 
> developers of a project about whether an API should continue to be 
> supported, then clearly, the Python developers get to win.  Welcome to 
> open source.
> 
> However, I believe that the core team is not paying attention to other 
> projects breaking.

Quite true. It is their duty to bring the breakage to our attention, if
they want to see results.

> I'm not saying that you want to make breaking 
> changes, or that bug reports are not dealt with, but the problem is that 
> dealing with these problems after the fact makes it _much_ more painful. 
> When you get to the release, and you have 30 bug reports due to other 
> projects breaking, they get triaged, some get left in, and some features 
> of lots of different projects are left broken.  And many projects do not 
> bother to test with the beta.

I thought we're talking about the projects that *do* use the buildbots?

> If developers were simply presented with the results of their changes 
> immediately ("you broke twisted, django doesn't import, zope segfaults 
> and mercurial destroys your repository") then perhaps they would weigh 
> the benefits of compatibility differently, and do the trivial, obvious 
> fix immediately, rather than waiting 6 months and having to figure out 
> what the heck change caused the bug. 

This sounds very nice in theory, but it must be implemented in a way that
does not add a burden to the developer, as Martin said.

 > Case in point: changes to the warnings module being discussed in a
 > separate thread break a significant number of Twisted's tests, because
 > they remove functionality which is not public but is required to test
 > code that uses the warnings module.  Would Brett have taken this into
 > account if he knew about it immediately when revision 62303 was
 > committed?  Maybe not, but now that the code is written and done,
 > there's significantly less motivation to go back and fix it.  At the
 > time it might have only been a few minutes' work to continue supporting
 > the use-case which Twisted needs.  Brett wouldn't even have necessarily
 > needed to do the work: it would have been easier to convince a Twisted
 > developer to do the work it to keep the community buildbot green rather
 > than to make it a bit less red.  Now, though, it's much easier to say
 > "oh well, that's private, you lose".  I don't ascribe this to malice -
 > it really *would* be much harder to fix it now, for us as well as for
 > him.

I must say that I'm increasingly surprised by this discussion since it seems
that not only the Python core devs neglect the community buildbots.
Shouldn't the project authors have at least the same interest that their
code runs on the next major Python version? And while they don't necessarily
have more time to watch the buildbots than we have, the amount of what they
need to keep an eye on.

Georg