[python-committers] 4 weeks with the new workflow: what needs changing?

[Brett Cannon]

On Mon, 13 Mar 2017 at 09:48 Yury Selivanov <yselivanov.ml at gmail.com> wrote:

> Hi,
>
> First, I'm really happy that we moved to git and GH.  The GH review tool
> is super convenient and CI integration helps.
>
> I'm less happy about requiring to make a PR for every commit. It's a no
> problem for new features development, but it's a huge pain for a bug
> fixing workflow.  Last week I needed to push a bunch of bug fixes to
> 3.7/3.6/3.5.  I had about 6 pull requests to the master branch, + 12
> more for 3.6 and 3.5.  I basically killed my entire second half of the
> day waiting until CI checks clear out.  I spend a few hours pushing just
> 3 (!!) committs to all branches.  Thanks to Brett, who disabled the push
> CI check for a while, I was able to push the remaining 3 bug fixes and
> their cherry picks in under 10 minutes.
>
> Yesterday I was working on a few asyncio PRs and a bug in async/await.
> All PRs required cherry-picking.  Again, I was spending significant
> amount of time just creating branches/PRs for cherry-picking.


Were you creating the branches manually or using Mariatta's script?


>   Again
> waiting for CI checks (even though I always run the test suite before I
> push).  In the end of the day, I was so frustrated and discouraged that
> I just stopped working on CPython.
>

Our Travis runs just got increased today so this should be improved. As I
have also previously said we can consider scaling back the number of things
we're building if necessary (i.e. we could go as low as 3 instead of the 5
we currently have or trying building using g++ to combine gcc and the C++
header check).


>
> The current workflow makes it significantly harder for me to be
> productive.  At this point I'm so discouraged of working on any bug
> fixes that I consider to stop working on them until the full
> cherry-picking automation is implemented.
>
> So I guess what I'm asking for is to consider turning the CI/PR push
> requirement off until we have automatic cherry-picking and a new NEWS
> file workflow.  We lived without this check for many years with
> mercurial.  Yes, all of us pushed some broken commits, but we have
> buildbots and CI, so nothing horrible ever happended.
>

It's a double-edged sword when you don't gate on CI but you have it for all
PRs. When Serhiy accidentally broke the build when I turned off Travis
gating for you, all PRs for a few hours came in as failing and it confused
some external contributors as to why their PR was coming up red. I had to
tell people that it was already broken before their PR was submitted and
once the fix was in that they needed to rebase to make sure their tests
pass. And the only way to make this not be an issue without gating on CI is
to require all PRs to be fully rebased before merging which is a constant
merge race (which we all know from our forward merging days to be a massive
pain and would then also require all PRs be merged through the command-line
and never online in order to do the rebase for contributors who would
always be behind).

IOW having CI is somewhat of an all-or-nothing thing here, else you are
dealing with the fallout of a broken build for a while (compared to the
Mercurial days where we all did the rebasing manually along with testing in
order to avoid this issue). Now as Donald pointed out, our concurrency
levels went up this morning so hopefully that will help with this.

It also doesn't help when randomness schedules test_tools, test_datetime,
or test_tokenize towards the end of a test run since they all take a few
minutes under clang (don't know why specifically clang, but there you go).

My point is that there are still some things we can do to make the
turn-around time on CI to be faster: see if the slower tests could be sped
up, deciding if we need all builds that we currently have, seeing if our
new concurrency levels help, or even consider gating on AppVeyor instead of
Travis. If all of those things are tried and we are still seeing
unacceptably long wait times on PRs, then we can talk about turning off the
CI gating. Does that seem fair?

-Brett


>
> Thank you,
> Yury
>
>
> On 2017-03-10 5:13 PM, Brett Cannon wrote:
> > I can't believe it's been 4 weeks. It feels like it was ages/yesterday
> > when we moved. :)
> >
> > First, I hope people are not regretting letting/having me make this
> > migration. I know there have been some things to work through (and
> > others still to come), but I hope this is all a net positive (either
> > now or in the near future).
> >
> > Second, I wanted to get initial feedback on things we can easily tweak:
> >
> >   * Requiring Travis to pass (I *really* don't want to turn this off
> >     as we already had a broken build when I temporarily turned it off
> >     at someone's request when Travis was backed up from the AWS S3
> >     outage; I also don't plan to make AppVeyor required unless there's
> >     a way to make it be skipped for doc-only changes)
> >   * Cherry-picking working out? (We can go back to forward merging if
> >     people really want to, but I think long-term cherry-picking will
> >     allow for more automation)
> >       o Along with that, are the labels for cherry-picking working
> >         out? (Some devs seem to like using title labels like `[3.6]`
> >         to flag cherry-picks so it's more obvious in emails so I don't
> >         know if the labels are really that useful)
> >   * Is the mention bot helpful? (Our config is at
> >     https://github.com/python/cpython/blob/master/.mention-bot and the
> >     docs are at https://github.com/facebook/mention-bot)
> >   * Anything to tweak about the coverage bot and reports? (Our config
> >     is at
> >     https://github.com/python/cpython/blob/master/.codecov.yml and
> >     docs at https://docs.codecov.io/docs/codecov-yaml)
> >
> > Third, I wanted to point out some of the more critical discussions
> > going on at https://github.com/python/core-workflow/issues.
> > Specifically, https://github.com/python/core-workflow/issues/6 is
> > working towards a solution for Misc/NEWS so if you care about the
> > final solution you should participate there. After Misc/NEWS is solved
> > the next step becomes solving the cherry-picking overhead with a more
> > automated approach. We are also discussing closed branches to make the
> > list of branches more manageable at
> > https://github.com/python/core-workflow/issues/31.
> >
> > Fourth, the lack of messages showing up on bugs.python.org
> > <http://bugs.python.org> after a commit is being tracked at
> > http://psf.upfronthosting.co.za/roundup/meta/issue613. I'm sure Ezio
> > and Maciej would appreciate any help people may be able to volunteer
> > to help in solving the problem.
> >
> > Fifth, anything I missed? :)
> >
> >
> > _______________________________________________
> > python-committers mailing list
> > python-committers at python.org
> > https://mail.python.org/mailman/listinfo/python-committers
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>
> _______________________________________________
> python-committers mailing list
> python-committers at python.org
> https://mail.python.org/mailman/listinfo/python-committers
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-committers/attachments/20170314/7f19c627/attachment-0001.html>