[python-committers] Revert changes which break too many buildbots

[Nick Coghlan]

On 15 June 2017 at 00:40, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> The CPython workflow was enhanced to get pre-commit CI checks. That's
> a huge win, thank you for that... But, sometimes, a change can still
> break many buildbots, bugs which weren't catched by pre-commit checks
> (Travis CI/Linux and AppVeyor/Windows). Buildbots cover much more
> different architectures and platforms.
>
> I spend a significant amount of time to maintain the sanity of our
> buildbots. Sometimes, it can take me up to 3 days on a week (of 5
> working days). It's annoying to see new regressions while I'm trying
> hard to fix old ones :-(
>
> So I would like to set a new rule: if I'm unable to fix buildbots
> failures caused by a recent change quickly (say, in less than 2
> hours), I propose to revert the change.

I'm not necessarily opposed to such a policy change, but if folks
really want guaranteed green post-merge buildbots for all platforms
(rather than just guaranteed green for Linux & Windows, sometimes red
for everything else), then I think a better place to focus effort
would be in making it easier for us to test things across the full
BuildBot fleet in pre-merge CI.

For example, something that would be genuinely helpful would be a bot
monitoring PR comments that could automate the "custom-build" dance
for core developers (i.e. we'd be able to write something like
"BuildBot: test custom build", and it would go away, kick off a custom
BuildBot run by pushing the PR to the "custom-build" branch, and then
report back the links for failed builds like
http://buildbot.python.org/all/builders/x86%20Tiger%20custom/builds/5
)

That way, the reversion process would be:

1. Revert the change
2. Post a "BuildBot: test custom build" comment on the offending PR
3. Original PR author, committer, and anyone else interested continues
the issue resolution process based on the specific links posted back
by the helper bot

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia