From brett at python.org Fri Jan 1 00:19:36 2016 From: brett at python.org (Brett Cannon) Date: Fri, 01 Jan 2016 05:19:36 +0000 Subject: [core-workflow] Longer term idea: consider Rust's homu for CPython merge gating In-Reply-To: References: Message-ID: Larry Hastings also wrote one that's attached to the issue regarding fixing the NEWS merge problem. On Thu, 31 Dec 2015, 20:47 Nick Coghlan wrote: > On 1 January 2016 at 06:21, Brett Cannon wrote: > > Thanks for the link! For those that didn't look at the link, Homu is > > actually written in Python and not Rust (the website for the project is > > what's written in Rust). > > > > Berker has actually said he is beginning to look into writing a bot for > our > > needs so he might be interested in this. > > > > A quick perusal suggests it probably needs to be modified to understand > > supporting multiple branches. I also don't know if it supports the > > fast-forwarding/rebasing commits we want, but I doubt that would be > > difficult to add. It probably also needs to grow an extension system > somehow > > to support custom NEWS entries for us. > > Which reminds me, Amber Brown recently wrote a NEWS file generator > based on the file-per-issue model, later assembled into a NEWS file > section as part of the project's release process: > https://github.com/hawkowl/towncrier > > It's aimed more at projects with a simple linear release history > rather than CPython's branching sprawl, though. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jan 1 14:21:05 2016 From: brett at python.org (Brett Cannon) Date: Fri, 01 Jan 2016 19:21:05 +0000 Subject: [core-workflow] We will be moving to GitHub Message-ID: I don't think this will be a shock to anyone who has followed the discussion on this list. The decision is essentially based on: 1. No major distinguishing features between GitHub or GitLab 2. Familiarity amongst core devs -- and external contributors -- with GitHub 3. Guido prefers GitHub Neither platform had some mind-blowing feature(s) that really made them stand out from each other such that it would greatly simplify our lives if we chose one platform over another. I obviously was really hoping there was going to be something I missed, but nothing ever came up (and no, being open source is not enough of a feature; as I said when I started this process, being open source would help break ties or minor lead of one tool but not be a deciding factor). But what Github does have over GitLab is familiarity. While there were people who publicly said they would prefer not to go with GitHub but would begrudgingly use it if we chose to go that route, I had multiple core devs email me privately saying they hoped I would choose GitHub. I think most of that stemmed from having used GitHub for other open source projects and/or work, making even dormant core devs say they would be able to become active again if we switched to GitHub thanks to eliminating the barrier of having to keep up with our custom workflow for code reviews and using hg for commits. And while I said it wasn't a goal to make things easier for external contributors, I also can't ignore the fact that the vast majority of people out there who might want to help out are already familiar with GitHub. And at least for me, the fact Guido prefers GitHub means something. While Guido himself would say I shouldn't really worry about his preferences since he is only an occasional contributor at this point, I believe that it's important that our BDFL actually like contributing to his own programming language rather than potentially alienating him because he finds the process burdensome. So that's why I have chosen GitHub over GitLab. Please realize that this is choosing GitHub to provide repository hosting and code review; we are not moving our issue tracker, nor are we moving our wiki. And the long-term plan is to set up a bot that will handle our commit workflow which will help isolate us from any repository hosting platform we are on and making moving easier in the future (and short-term people will use the command-line and that's totally platform-agnostic). Thanks to everyone who contributed to this decision, especially Donald, Barry, and Nick for making the proposals we had to work from. We can start the discussion of how we want to handle the transition next week, but I'm going to try and step away from this whole workflow topic until Monday so I can spend the last couple of days of my vacation not thinking about this stuff. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skrah.temporarily at gmail.com Fri Jan 1 15:25:11 2016 From: skrah.temporarily at gmail.com (Stefan Krah) Date: Fri, 1 Jan 2016 20:25:11 +0000 (UTC) Subject: [core-workflow] We will be moving to GitHub References: Message-ID: Brett Cannon writes: > I don't think this will be a shock to anyone who has followed the discussion on this list. The decision is essentially based on: We must have been reading different discussions: On *this* list more people were in favor of GitLab! Except for Guido, Donald and Senthil (on python-committers) no one has even bothered to post an explicit +1 for GitHub. It's very disappointing that GitHub proponents apparently felt the need to contact you privately instead of stating their opinions in the open. Stefan Krah From rdmurray at bitdance.com Fri Jan 1 16:30:43 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 01 Jan 2016 16:30:43 -0500 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: <20160101213045.EB0B6B1408B@webabinitio.net> On Fri, 01 Jan 2016 20:25:11 +0000, Stefan Krah wrote: > Brett Cannon writes: > > I don't think this will be a shock to anyone who has followed the > discussion on this list. The decision is essentially based on: > > We must have been reading different discussions: On *this* list more > people were in favor of GitLab! Except for Guido, Donald and Senthil > (on python-committers) no one has even bothered to post an explicit > +1 for GitHub. > > It's very disappointing that GitHub proponents apparently felt the > need to contact you privately instead of stating their opinions in > the open. The impression I got here was Barry advocating GitLab (of course), and you arguing against GitHub (but not, particularly, in favor of GitLab). Everything else was down at the noise level :) Me, I don't care one way or the other, as long as we aren't locking ourselves in to either. Now, the fact that people felt it better to contact Brett privately to advocate for GitHub is indeed interesting, and yes, disappointing. The interesting question is, why is that? Perhaps it is what was alluded to earlier, that favoring the "commercial alternative" is seen as "bad" in terms of what we might label as "virtue signalling"? Which would be weird, because GitLab isn't non-commercial. So maybe there's some other reason (because GitHub is the big gorilla and people think it is "better" to favor the underdog?), but I wonder if it still comes down to virtue signalling (or, rather, not wanting to signal non-virtue, in this case). So I agree with you, it would be great if people would openly speak their minds, as Guido did :) On the other hand, it might just be a matter of the "usenet nod", and not wanting to "clutter up the list" with a "me too". You did get some pushback against your arguments, to which people may have been nodding. --David From ncoghlan at gmail.com Fri Jan 1 20:17:24 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 2 Jan 2016 11:17:24 +1000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: <20160101213045.EB0B6B1408B@webabinitio.net> References: <20160101213045.EB0B6B1408B@webabinitio.net> Message-ID: On 2 Jan 2016 07:37, "R. David Murray" wrote: > > On Fri, 01 Jan 2016 20:25:11 +0000, Stefan Krah < skrah.temporarily at gmail.com> wrote: > > Brett Cannon writes: > > > I don't think this will be a shock to anyone who has followed the > > discussion on this list. The decision is essentially based on: > > > > We must have been reading different discussions: On *this* list more > > people were in favor of GitLab! Except for Guido, Donald and Senthil > > (on python-committers) no one has even bothered to post an explicit > > +1 for GitHub. > > > > It's very disappointing that GitHub proponents apparently felt the > > need to contact you privately instead of stating their opinions in > > the open. > > The impression I got here was Barry advocating GitLab (of course), and > you arguing against GitHub (but not, particularly, in favor of GitLab). > Everything else was down at the noise level :) > > Me, I don't care one way or the other, as long as we aren't locking > ourselves in to either. > > Now, the fact that people felt it better to contact Brett privately to > advocate for GitHub is indeed interesting, and yes, disappointing. The > interesting question is, why is that? Perhaps it is what was alluded to > earlier, that favoring the "commercial alternative" is seen as "bad" in > terms of what we might label as "virtue signalling"? Which would be > weird, because GitLab isn't non-commercial. So maybe there's some other > reason (because GitHub is the big gorilla and people think it is > "better" to favor the underdog?), but I wonder if it still comes down to > virtue signalling (or, rather, not wanting to signal non-virtue, in this > case). > > So I agree with you, it would be great if people would openly speak > their minds, as Guido did :) > > On the other hand, it might just be a matter of the "usenet nod", and > not wanting to "clutter up the list" with a "me too". You did get some > pushback against your arguments, to which people may have been nodding. > > --David > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Fri Jan 1 20:57:32 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Sat, 2 Jan 2016 03:57:32 +0200 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: Hi, On Fri, Jan 1, 2016 at 9:21 PM, Brett Cannon wrote: > I don't think this will be a shock to anyone who has followed the discussion > on this list. The decision is essentially based on: > > No major distinguishing features between GitHub or GitLab > Familiarity amongst core devs -- and external contributors -- with GitHub > Guido prefers GitHub > > Neither platform had some mind-blowing feature(s) that really made them > stand out from each other such that it would greatly simplify our lives if > we chose one platform over another. I obviously was really hoping there was > going to be something I missed, but nothing ever came up (and no, being open > source is not enough of a feature; as I said when I started this process, > being open source would help break ties or minor lead of one tool but not be > a deciding factor). > > But what Github does have over GitLab is familiarity. While there were > people who publicly said they would prefer not to go with GitHub but would > begrudgingly use it if we chose to go that route, I had multiple core devs > email me privately saying they hoped I would choose GitHub. I think most of > that stemmed from having used GitHub for other open source projects and/or > work, making even dormant core devs say they would be able to become active > again if we switched to GitHub thanks to eliminating the barrier of having > to keep up with our custom workflow for code reviews and using hg for > commits. And while I said it wasn't a goal to make things easier for > external contributors, I also can't ignore the fact that the vast majority > of people out there who might want to help out are already familiar with > GitHub. > > And at least for me, the fact Guido prefers GitHub means something. While > Guido himself would say I shouldn't really worry about his preferences since > he is only an occasional contributor at this point, I believe that it's > important that our BDFL actually like contributing to his own programming > language rather than potentially alienating him because he finds the process > burdensome. > > So that's why I have chosen GitHub over GitLab. Please realize that this is > choosing GitHub to provide repository hosting and code review; we are not > moving our issue tracker, nor are we moving our wiki. And the long-term plan > is to set up a bot that will handle our commit workflow which will help > isolate us from any repository hosting platform we are on and making moving > easier in the future (and short-term people will use the command-line and > that's totally platform-agnostic). > > Thanks to everyone who contributed to this decision, especially Donald, > Barry, and Nick for making the proposals we had to work from. > > We can start the discussion of how we want to handle the transition next > week, but I'm going to try and step away from this whole workflow topic > until Monday so I can spend the last couple of days of my vacation not > thinking about this stuff. :) > This will likely require a new PEP, that should cover: 1) the new workflow (including how to handle reviews -- see below); 2) the steps required for the migration and a timeline; 3) a list of things that will break and/or that will need to be added/replaced before/during/after the migration; 4) the fate of hg.python.org; Some of these things might already be covered by existing PEPs, but I don't see them in PEP 507[0] and 481[1] (and I'm getting a bit lost among all the competing PEPs and multiple threads across at least a couple different MLs :). Above you said that GitHub will be used for reviews but we will keep using our bug tracker. This leads to the following questions: * Do GitHub reviews only work for pull requests? * Are we still going to support uploading diffs/patches to the tracker (short term and long term)? * If so, how/where are we going to review those diffs/patches? * Will patches be automatically converted to pull requests, or pull requests converted to patches and attached to b.p.o issues? * Are there plans to migrate existing Rietveld reviews to GitHub and shut Rietveld down for good? * If not, will Rietveld stay around and be read-only? Or will it still be used for patches uploaded to b.p.o? * What else is needed to integrate GitHub and b.p.o? About points 3 of the initial list, these are some examples of things that will need to be added/updated/replaced: * the hgroundup hook[2] that post messages to b.p.o when a commit includes an issue number needs to be replaced; * the hgirker hook[3] used for deadparrot on #python-dev needs to be replaced (probably there is already an irker-git hook that can be used/adapted); * the hgbuildbot[4] hook that triggers the buildbots on commit needs to be replaced; * other hg hooks[5] might need to be rewritten/replaced; * the script[6] that generates bug tracker links to hg.p.o needs to be updated (for both cs ids and paths); * the hg code that converts issue links in commit messages to b.p.o links needs to be replaced; * other places where issue numbers appears on GitHub should generate links to b.p.o; * the hg-cpydev hg extension [7] should be ported to git (optional); * the buildbots need to be updated if they are going to pull the source from github and use git; * the bug tracker will need to be updated to interact with github; * the devguide needs to be updated (both to cover the new workflow and update links/commands); FWIW next weekend (9-10 January) we are organizing a sprint in Helsinki, and I'm planning to work on the bug tracker, so I might be able to start addressing some of these points. Best Regards, Ezio Melotti P.S. enjoy your last few days of vacation while you can ;) [0]: https://www.python.org/dev/peps/pep-0507/ [1]: https://www.python.org/dev/peps/pep-0481/ [2]: https://hg.python.org/hooks/file/tip/hgroundup.py [3]: https://hg.python.org/hooks/file/tip/hgirker.py [4]: https://hg.python.org/hooks/file/tip/hgbuildbot.py [5]: https://hg.python.org/hooks/file/tip [6]: https://hg.python.org/tracker/python-dev/file/tip/extensions/local_replace.py [7]: https://bitbucket.org/introom/hg-cpydev From ncoghlan at gmail.com Fri Jan 1 21:07:19 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 2 Jan 2016 12:07:19 +1000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: <20160101213045.EB0B6B1408B@webabinitio.net> Message-ID: (Sorry, accidentally hit send while trying to discard a previous draft) On 2 Jan 2016 11:17, "Nick Coghlan" wrote: > On 2 Jan 2016 07:37, "R. David Murray" wrote: > > Now, the fact that people felt it better to contact Brett privately to > > advocate for GitHub is indeed interesting, and yes, disappointing. The > > interesting question is, why is that? Perhaps it is what was alluded to > > earlier, that favoring the "commercial alternative" is seen as "bad" in > > terms of what we might label as "virtue signalling"? Which would be > > weird, because GitLab isn't non-commercial. So maybe there's some other > > reason (because GitHub is the big gorilla and people think it is > > "better" to favor the underdog?), but I wonder if it still comes down to > > virtue signalling (or, rather, not wanting to signal non-virtue, in this > > case). Yep, I think that's a large part of it, as the folks funding GitHub are quite open about the fact that they consider centralised US corporate control of the technology industry something to be admired, rather than deplored (See http://techcrunch.com/2014/02/13/please-dont-tell-me-you-want-to-be-the-next-red-hat/ for one of the most explicit examples of that genre). GitLab's business model is different, as it's just a low cost competitor in the self-hosted VCS market that takes advantage of people's familiarity with GitHub's UI, rather than aiming to become the de facto standard for open collaboration infrastructure (with corresponding influence over the career prospects of individual developers). The free software movement has been fighting an underdog battle against that kind of centralised control for 30 years. That hasn't been a stellar success so far, with the likes of Amazon, Google, Facebook, Apple and Microsoft v2.0 now making their predecessors like IBM, Oracle and Microsoft v1.0 look like rank amateurs when it comes to exerting centralised control over the world's computing infrastructure. (Tom Watson's apocryphal prediction of a world market for maybe 5 computers seems likely to come true, it's just that their names will be AWS, Azure, GCE, Aliyun, and SoftLayer). That doesn't explain why folks might be reluctant to state a preference for a proprietary service in public, though. For that, I think we need to account for the fact that the free software movement is often it's own worst enemy, as it tends to be *very* focused on ideological purity, so proprietary dependencies are considered categorically unacceptable in many circles, rather than as risks to be mitigated through pragmatic measures. Trying to explain "We looked the gift horse in the mouth, checked all its teeth, and are happy we can deal with the risks of accepting the gift" can be incredibly draining when you have folks yelling at you that accepting gratis contributions of proprietary software and services mean you don't care about the future of the open source community or about software freedom. I stepped over that line myself back when the GitHub proposal was first put forward, and Guido quite rightly called me on it - I wasn't properly separating my own long term objectives from the immediate interests of the CPython core development community. Since the pro-GitHub perspective was being suitably represented already (and was clearly the default choice, with most of the discussions focusing on "Is there a reason to *not* just use GitHub?"), why *would* anyone want to risk exposing themselves to that kind of potential negative response? Cheers, Nick. From francismb at email.de Sat Jan 2 09:55:54 2016 From: francismb at email.de (francismb) Date: Sat, 2 Jan 2016 15:55:54 +0100 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: <5687E4FA.6020507@email.de> Dear Cores-Devs, > > [...] And the long-term > plan is to set up a bot that will handle our commit workflow which will > help isolate us from any repository hosting platform we are on and making > moving easier in the future (and short-term people will use the > command-line and that's totally platform-agnostic). > please let us known wen/were that bot/automation is going to take place. I've interest on that and I would like to participate :-). Thanks in advance and have a prosperous new year! francis From willingc at willingconsulting.com Sat Jan 2 12:44:14 2016 From: willingc at willingconsulting.com (Carol Willing) Date: Sat, 02 Jan 2016 09:44:14 -0800 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: <56880C6E.2070502@willingconsulting.com> Thank you Brett, Donald, Barry, Nick, Ezio, and all who have worked on core workflow. I believe that moving code review to GitHub will benefit CPython and allow more automated checks and feedback (such as those provided by Travis, codecov, coverage, etc) that will help new contributors as well as save core developers' time. Ezio, Great post on details to think about. Please keep me looped in on issue tracker enhancements (which I know we are not moving) that may make the issue tracker more convenient and appealing to users (similar to PyPi migration to Warehouse). Brett, Enjoy the rest of your well earned vacation. All, Happy New Year and all the best for 2016. Warmly, Carol > Brett Cannon > January 1, 2016 at 11:21 AM > I don't think this will be a shock to anyone who has followed the > discussion on this list. The decision is essentially based on: > > 1. No major distinguishing features between GitHub or GitLab > 2. Familiarity amongst core devs -- and external contributors -- with > GitHub > 3. Guido prefers GitHub > > Neither platform had some mind-blowing feature(s) that really made > them stand out from each other such that it would greatly simplify our > lives if we chose one platform over another. I obviously was really > hoping there was going to be something I missed, but nothing ever came > up (and no, being open source is not enough of a feature; as I said > when I started this process, being open source would help break ties > or minor lead of one tool but not be a deciding factor). > > But what Github does have over GitLab is familiarity. While there were > people who publicly said they would prefer not to go with GitHub but > would begrudgingly use it if we chose to go that route, I had multiple > core devs email me privately saying they hoped I would choose GitHub. > I think most of that stemmed from having used GitHub for other open > source projects and/or work, making even dormant core devs say they > would be able to become active again if we switched to GitHub thanks > to eliminating the barrier of having to keep up with our custom > workflow for code reviews and using hg for commits. And while I said > it wasn't a goal to make things easier for external contributors, I > also can't ignore the fact that the vast majority of people out there > who might want to help out are already familiar with GitHub. > > And at least for me, the fact Guido prefers GitHub means something. > While Guido himself would say I shouldn't really worry about his > preferences since he is only an occasional contributor at this point, > I believe that it's important that our BDFL actually like contributing > to his own programming language rather than potentially alienating him > because he finds the process burdensome. > > So that's why I have chosen GitHub over GitLab. Please realize that > this is choosing GitHub to provide repository hosting and code review; > we are not moving our issue tracker, nor are we moving our wiki. And > the long-term plan is to set up a bot that will handle our commit > workflow which will help isolate us from any repository hosting > platform we are on and making moving easier in the future (and > short-term people will use the command-line and that's totally > platform-agnostic). > > Thanks to everyone who contributed to this decision, especially > Donald, Barry, and Nick for making the proposals we had to work from. > > We can start the discussion of how we want to handle the transition > next week, but I'm going to try and step away from this whole workflow > topic until Monday so I can spend the last couple of days of my > vacation not thinking about this stuff. :) > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct -- Sent from Postbox -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jan 2 13:19:32 2016 From: brett at python.org (Brett Cannon) Date: Sat, 02 Jan 2016 18:19:32 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: <20160101213045.EB0B6B1408B@webabinitio.net> References: <20160101213045.EB0B6B1408B@webabinitio.net> Message-ID: On Fri, 1 Jan 2016 at 13:37 R. David Murray wrote: > On Fri, 01 Jan 2016 20:25:11 +0000, Stefan Krah < > skrah.temporarily at gmail.com> wrote: > > Brett Cannon writes: > > > I don't think this will be a shock to anyone who has followed the > > discussion on this list. The decision is essentially based on: > > > > We must have been reading different discussions: On *this* list more > > people were in favor of GitLab! Except for Guido, Donald and Senthil > > (on python-committers) no one has even bothered to post an explicit > > +1 for GitHub. > > > > It's very disappointing that GitHub proponents apparently felt the > > need to contact you privately instead of stating their opinions in > > the open. > > The impression I got here was Barry advocating GitLab (of course), and > you arguing against GitHub (but not, particularly, in favor of GitLab). > Everything else was down at the noise level :) > That's the impression I got as well. The reason I didn't think it would be surprising was my constant prodding for GitLab features that made it stand out. I wouldn't have bothered pushing for that if I had already chosen GitLab or thought it was on track to win out. The fact that I persistently tried to make the open source solution somehow shine was because I was trying to grasp on to something for GitLab because I was trying to avoid the stress of dealing with people being unhappy with selecting GitHub by making my decision glaringly obvious (and in all honesty it would have been nice if an open source solution could have won out; maybe in the future). In the end, though, I decided dealing with people upset from going with GitHub was worth it compared to the positives of selecting GitHub, hence my decision. > > Me, I don't care one way or the other, as long as we aren't locking > ourselves in to either. > > Now, the fact that people felt it better to contact Brett privately to > advocate for GitHub is indeed interesting, and yes, disappointing. The > interesting question is, why is that? Perhaps it is what was alluded to > earlier, that favoring the "commercial alternative" is seen as "bad" in > terms of what we might label as "virtue signalling"? Which would be > weird, because GitLab isn't non-commercial. So maybe there's some other > reason (because GitHub is the big gorilla and people think it is > "better" to favor the underdog?), but I wonder if it still comes down to > virtue signalling (or, rather, not wanting to signal non-virtue, in this > case). > > So I agree with you, it would be great if people would openly speak > their minds, as Guido did :) > > On the other hand, it might just be a matter of the "usenet nod", and > not wanting to "clutter up the list" with a "me too". You did get some > pushback against your arguments, to which people may have been nodding. > The comments came of two forms. Some were of "yay for GitHub" so basically a one sentence head nod. The other was a few private messages that were a bit longer and explaining their view on the options (including not switching). The vast majority of these emails, though, were in reaction to my email to python-committers asking for people to tell me if they would walk away if we chose GitHub. Most said they were emailing me privately to avoid adding any unnecessary noise to that email thread, but I suspect it also had to do with them not being subscribers to core-workflow and not caring enough to join this list to wade into the discussion. But also realize that this process has been going on for over a year now. I have had multiple conversations at conferences at this point with people who expressed various opinions on the matter and I didn't report those face-to-face conversations either and which are no different than a private email. There was never going to be a chance where anyone but me was going to have complete knowledge of people's various positions, nor was I about to report a tally of those conversations based on preferences. I'm sorry if people feel like I did a disservice by not doing a regular "general sentiment of the Python community" report based on what private feedback I received, but I just didn't think about it nor did I think it would be an issue for anyone that people chose to speak with me privately. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jan 2 13:45:00 2016 From: brett at python.org (Brett Cannon) Date: Sat, 02 Jan 2016 18:45:00 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: I was purposefully trying to avoid having this discussion start until Monday, but since Ezio sort has a deadline for issue-related stuff we can start on that side now (I still have an email planned on Monday to outline the initial steps to moving things over and the very first hurdle to work through). On Fri, 1 Jan 2016 at 17:57 Ezio Melotti wrote: > Hi, > > On Fri, Jan 1, 2016 at 9:21 PM, Brett Cannon wrote: > > I don't think this will be a shock to anyone who has followed the > discussion > > on this list. The decision is essentially based on: > > > > No major distinguishing features between GitHub or GitLab > > Familiarity amongst core devs -- and external contributors -- with GitHub > > Guido prefers GitHub > > > > Neither platform had some mind-blowing feature(s) that really made them > > stand out from each other such that it would greatly simplify our lives > if > > we chose one platform over another. I obviously was really hoping there > was > > going to be something I missed, but nothing ever came up (and no, being > open > > source is not enough of a feature; as I said when I started this process, > > being open source would help break ties or minor lead of one tool but > not be > > a deciding factor). > > > > But what Github does have over GitLab is familiarity. While there were > > people who publicly said they would prefer not to go with GitHub but > would > > begrudgingly use it if we chose to go that route, I had multiple core > devs > > email me privately saying they hoped I would choose GitHub. I think most > of > > that stemmed from having used GitHub for other open source projects > and/or > > work, making even dormant core devs say they would be able to become > active > > again if we switched to GitHub thanks to eliminating the barrier of > having > > to keep up with our custom workflow for code reviews and using hg for > > commits. And while I said it wasn't a goal to make things easier for > > external contributors, I also can't ignore the fact that the vast > majority > > of people out there who might want to help out are already familiar with > > GitHub. > > > > And at least for me, the fact Guido prefers GitHub means something. While > > Guido himself would say I shouldn't really worry about his preferences > since > > he is only an occasional contributor at this point, I believe that it's > > important that our BDFL actually like contributing to his own programming > > language rather than potentially alienating him because he finds the > process > > burdensome. > > > > So that's why I have chosen GitHub over GitLab. Please realize that this > is > > choosing GitHub to provide repository hosting and code review; we are not > > moving our issue tracker, nor are we moving our wiki. And the long-term > plan > > is to set up a bot that will handle our commit workflow which will help > > isolate us from any repository hosting platform we are on and making > moving > > easier in the future (and short-term people will use the command-line and > > that's totally platform-agnostic). > > > > Thanks to everyone who contributed to this decision, especially Donald, > > Barry, and Nick for making the proposals we had to work from. > > > > We can start the discussion of how we want to handle the transition next > > week, but I'm going to try and step away from this whole workflow topic > > until Monday so I can spend the last couple of days of my vacation not > > thinking about this stuff. :) > > > > This will likely require a new PEP, that should cover: > 1) the new workflow (including how to handle reviews -- see below); > 2) the steps required for the migration and a timeline; > 3) a list of things that will break and/or that will need to be > added/replaced before/during/after the migration; > 4) the fate of hg.python.org; > Yes to all of that. > > Some of these things might already be covered by existing PEPs, but I > don't see them in PEP 507[0] and 481[1] (and I'm getting a bit lost > among all the competing PEPs and multiple threads across at least a > couple different MLs :). > It will be either be a new PEP or the GitHub PEP will simply be rewritten. > > Above you said that GitHub will be used for reviews but we will keep > using our bug tracker. > Yes, we are not moving to GitHub's issue tracker. > This leads to the following questions: > * Do GitHub reviews only work for pull requests? > What do you mean by this? Do you mean by patch upload? > * Are we still going to support uploading diffs/patches to the > tracker (short term and long term)? > Well, "support" as in "allow". We won't be keeping Rietveld around (part of this move is so we can get off of Rietveld). > * If so, how/where are we going to review those diffs/patches? > I don't know. Is it possible to have the bot create a PR for an uploaded patch? > * Will patches be automatically converted to pull requests, or pull > requests converted to patches and attached to b.p.o issues? > I don't think we need to upload a patch file if we already have a PR, but we should either have a PR -> issue or issue -> PR mapping somehow. This can either be via bot command or specifying a PR # in the issue tracker. I also have no issue if people want to make PRs generate patches for the issue as well for backup purposes, but that might get a bit noisy for those following both an issue and a PR. > * Are there plans to migrate existing Rietveld reviews to GitHub and > shut Rietveld down for good? > Yes, there are plans to shut Rietveld down. I suspect we will leave it up until GitHub is running in an equivalent fashion to our current workflow, then we will have a deadline to work through any outstanding reviews and then close it up (and if we want to be thorough, set Rietveld so that it won't work on new issues and only pre-existing ones that already have a patch). > * If not, will Rietveld stay around and be read-only? Or will it > still be used for patches uploaded to b.p.o? > We are trying to cut down on the infrastructure we maintain, so I don't want Rietveld sticking around indefinitely. > * What else is needed to integrate GitHub and b.p.o? > Basically a way to map an issue to a PR (or vice-versa). Probably the simplest solution is to allow pasting in a GitHub PR URL or the PR # to make the association. The other option is for the bot to accept a command to make the association. No reason we can't have both, though. But either way we should have a way to connect PRs to issues. If a bot can create PRs from a patch, then that would be nice for folks who don't want to use GitHub, but I don't consider that a priority. I'm happy to try to be accommodating to people who don't want to use GitHub for whatever reason, there is a limit to how much energy we should put into supporting that scenario past uploading a patch. > > About points 3 of the initial list, these are some examples of things > that will need to be added/updated/replaced: > * the hgroundup hook[2] that post messages to b.p.o when a commit > includes an issue number needs to be replaced; > * the hgirker hook[3] used for deadparrot on #python-dev needs to be > replaced (probably there is already an irker-git hook that can be > used/adapted); > * the hgbuildbot[4] hook that triggers the buildbots on commit needs > to be replaced; > * other hg hooks[5] might need to be rewritten/replaced; > * the script[6] that generates bug tracker links to hg.p.o needs to > be updated (for both cs ids and paths); > * the hg code that converts issue links in commit messages to b.p.o > links needs to be replaced; > * other places where issue numbers appears on GitHub should generate > links to b.p.o; > * the hg-cpydev hg extension [7] should be ported to git (optional); > * the buildbots need to be updated if they are going to pull the > source from github and use git; > * the bug tracker will need to be updated to interact with github; > * the devguide needs to be updated (both to cover the new workflow > and update links/commands); > Yes to all of these (I think; depends if we change how something operates in terms of associations). > > FWIW next weekend (9-10 January) we are organizing a sprint in > Helsinki, and I'm planning to work on the bug tracker, so I might be > able to start addressing some of these points. > I think deciding how to associate issues with PRs would be a great thing to work through. Otherwise we need to decide if we want to go with an issue tracker solution to the NEWS file or if we want a individual file solution (which happens to be bot-friendly). Those two things are probably the most critical starting points. -Brett > > Best Regards, > Ezio Melotti > > P.S. enjoy your last few days of vacation while you can ;) > > [0]: https://www.python.org/dev/peps/pep-0507/ > [1]: https://www.python.org/dev/peps/pep-0481/ > [2]: https://hg.python.org/hooks/file/tip/hgroundup.py > [3]: https://hg.python.org/hooks/file/tip/hgirker.py > [4]: https://hg.python.org/hooks/file/tip/hgbuildbot.py > [5]: https://hg.python.org/hooks/file/tip > [6]: > https://hg.python.org/tracker/python-dev/file/tip/extensions/local_replace.py > [7]: https://bitbucket.org/introom/hg-cpydev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jan 2 13:47:30 2016 From: brett at python.org (Brett Cannon) Date: Sat, 02 Jan 2016 18:47:30 +0000 Subject: [core-workflow] I'm going to let Jython know what's happening Message-ID: I don't know if they are going to ask to come under the python organization or go do their own thing, but I'm going to make sure they know what's going on over here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jan 2 14:13:28 2016 From: brett at python.org (Brett Cannon) Date: Sat, 02 Jan 2016 19:13:28 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: <5687E4FA.6020507@email.de> References: <5687E4FA.6020507@email.de> Message-ID: On Sat, 2 Jan 2016 at 07:01 francismb wrote: > Dear Cores-Devs, > > > > > [...] And the long-term > > plan is to set up a bot that will handle our commit workflow which will > > help isolate us from any repository hosting platform we are on and making > > moving easier in the future (and short-term people will use the > > command-line and that's totally platform-agnostic). > > > > please let us known wen/were that bot/automation is going to take place. > I've interest on that and I would like to participate :-). > Thanks for the offer to help! All of that will be discussed on this mailing list, so just stay subscribed and watch out for any details. -Brett > > Thanks in advance and have a prosperous new year! > francis > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skrah.temporarily at gmail.com Sat Jan 2 16:47:24 2016 From: skrah.temporarily at gmail.com (Stefan Krah) Date: Sat, 2 Jan 2016 21:47:24 +0000 (UTC) Subject: [core-workflow] We will be moving to GitHub References: <20160101213045.EB0B6B1408B@webabinitio.net> Message-ID: Brett Cannon writes: > But also realize that this process has been going on for over a year now. I have had multiple conversations at conferences at this point with people who expressed various opinions on the matter and I didn't report those face-to-face conversations either and which are no different than a private email. There was never going to be a chance where anyone but me was going to have complete knowledge of people's various positions, nor was I about to report a tally of those conversations based on preferences. I'm sorry if people feel like I did a disservice by not doing a regular "general sentiment of the Python community" report based on what private feedback I received, but I just didn't think about it nor did I think it would be an issue for anyone that people chose to speak with me privately. Thank you for the explanation. It's a bit of a recurring problem here that once something has been discussed at a conference, the people who weren't there have a very hard time to catch up via the mailing lists. So I don't think posting +-1 on python-committers is noise, in matters like this one it should happen more often. Sorry for starting this subthread during your vacation (I missed that), I hope you can enjoy the remaining days! Stefan Krah From ezio.melotti at gmail.com Sat Jan 2 17:35:14 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Sun, 3 Jan 2016 00:35:14 +0200 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: On Sat, Jan 2, 2016 at 8:45 PM, Brett Cannon wrote: > I was purposefully trying to avoid having this discussion start until > Monday, but since Ezio sort has a deadline for issue-related stuff we can > start on that side now (I still have an email planned on Monday to outline > the initial steps to moving things over and the very first hurdle to work > through). > No hurry, there's still a week before the sprint so take your time. I also still have to merge all the GSoC improvements before looking at these other issues. > On Fri, 1 Jan 2016 at 17:57 Ezio Melotti wrote: >> >> Hi, >> >> On Fri, Jan 1, 2016 at 9:21 PM, Brett Cannon wrote: >> > ... >> > We can start the discussion of how we want to handle the transition next >> > week, but I'm going to try and step away from this whole workflow topic >> > until Monday so I can spend the last couple of days of my vacation not >> > thinking about this stuff. :) >> > >> >> This will likely require a new PEP, that should cover: >> 1) the new workflow (including how to handle reviews -- see below); >> 2) the steps required for the migration and a timeline; >> 3) a list of things that will break and/or that will need to be >> added/replaced before/during/after the migration; >> 4) the fate of hg.python.org; > > > Yes to all of that. > >> >> >> Some of these things might already be covered by existing PEPs, but I >> don't see them in PEP 507[0] and 481[1] (and I'm getting a bit lost >> among all the competing PEPs and multiple threads across at least a >> couple different MLs :). > > > It will be either be a new PEP or the GitHub PEP will simply be rewritten. > OK. >> >> >> Above you said that GitHub will be used for reviews but we will keep >> using our bug tracker. > > > Yes, we are not moving to GitHub's issue tracker. > >> >> This leads to the following questions: >> * Do GitHub reviews only work for pull requests? > > > What do you mean by this? Do you mean by patch upload? > With reviews here I mean inline comments similar to the ones you can add on Rietveld. I haven't used GitHub yet, but I assume this can be done for pull requests (same as with BitBucket) and perhaps for patches attached to the GitHub bug tracker. Since we are not going to use the GitHub bug tracker, only pull requests can be reviewed on GitHub. If someone uploads a patch on the bug tracker it won't be possible to review it unless we convert it to a pull request (or unless we keep Rietveld around, but we don't want that). >> >> * Are we still going to support uploading diffs/patches to the >> tracker (short term and long term)? > > > Well, "support" as in "allow". We won't be keeping Rietveld around (part of > this move is so we can get off of Rietveld). > Only supporting pull requests will force all contributors to use GitHub (unless GitHub supports "external" pull requests). I think we should keep supporting plain patches as an alternative, and write tools to convert those to pull requests. If GitHub doesn't support "external" pull requests, we might also be able to write additional tools to do so from the tracker directly. >> >> * If so, how/where are we going to review those diffs/patches? > > > I don't know. Is it possible to have the bot create a PR for an uploaded > patch? > It should be possible -- we will need an account for b.p.o and then b.p.o can commit the patches on its clone (perhaps using a different branch for issue/patch) and send a pull request to the main clone (for the last bit we might need to use the GitHub API, unless there are other ways). >> >> * Will patches be automatically converted to pull requests, or pull >> requests converted to patches and attached to b.p.o issues? > > > I don't think we need to upload a patch file if we already have a PR, The goal is to have everything in the same place, even if contributions are made both via pull requests and via patches uploaded to b.p.o. I think the best option is to convert all patches uploaded to b.p.o to pull requests, so that all the reviews happen on GitHub. > but we > should either have a PR -> issue or issue -> PR mapping somehow. This can > either be via bot command or specifying a PR # in the issue tracker. b.p.o should automatically link to PRs when "PR num" appears in a message (this can be done easily). GitHub should also link to b.p.o issues when an issue number appears in the PR (I don't know if/how this can be done). For patches uploaded to b.p.o, a link to the generated PR can be added where the "review" link is now. > I also > have no issue if people want to make PRs generate patches for the issue as > well for backup purposes, but that might get a bit noisy for those following > both an issue and a PR. > I'm not sure we need to generate a diff and attach it to the issue, but PRs related to the issue should probably be listed on b.p.o where the list of patches currently is. One possible reason to add patches is the new patch analysis tool written during GSoC that can extract metadata from patches (affected files, whether the patch has tests/docs, etc.) which in turn can be used during searches. The tool can also be adapted to generate the diff dynamically and extract metadata from there though. >> >> * Are there plans to migrate existing Rietveld reviews to GitHub and >> shut Rietveld down for good? > > > Yes, there are plans to shut Rietveld down. I suspect we will leave it up > until GitHub is running in an equivalent fashion to our current workflow, Good to know. > then we will have a deadline to work through any outstanding reviews and > then close it up (and if we want to be thorough, set Rietveld so that it > won't work on new issues and only pre-existing ones that already have a > patch). > Or we could just copy-paste the reviews as messages on the tracker so that languishing reviews don't hold up the Rietveld demise. >> >> * If not, will Rietveld stay around and be read-only? Or will it >> still be used for patches uploaded to b.p.o? > > > We are trying to cut down on the infrastructure we maintain, so I don't want > Rietveld sticking around indefinitely. > Agreed. >> >> * What else is needed to integrate GitHub and b.p.o? > > > Basically a way to map an issue to a PR (or vice-versa). Probably the > simplest solution is to allow pasting in a GitHub PR URL or the PR # to make > the association. The other option is for the bot to accept a command to make > the association. No reason we can't have both, though. But either way we > should have a way to connect PRs to issues. > I'll have to think a bit about the best way to implement this and the UI (I already have some ideas). > If a bot can create PRs from a patch, then that would be nice for folks who > don't want to use GitHub, but I don't consider that a priority. I'm happy to > try to be accommodating to people who don't want to use GitHub for whatever > reason, there is a limit to how much energy we should put into supporting > that scenario past uploading a patch. > If we want to support this (and I think we should) we will either need a bot (actually a Roundup detector) or a human to do it, and the former sounds better. >> >> >> About points 3 of the initial list, these are some examples of things >> that will need to be added/updated/replaced: >> * the hgroundup hook[2] that post messages to b.p.o when a commit >> includes an issue number needs to be replaced; >> * the hgirker hook[3] used for deadparrot on #python-dev needs to be >> replaced (probably there is already an irker-git hook that can be >> used/adapted); >> * the hgbuildbot[4] hook that triggers the buildbots on commit needs >> to be replaced; >> * other hg hooks[5] might need to be rewritten/replaced; >> * the script[6] that generates bug tracker links to hg.p.o needs to >> be updated (for both cs ids and paths); >> * the hg code that converts issue links in commit messages to b.p.o >> links needs to be replaced; >> * other places where issue numbers appears on GitHub should generate >> links to b.p.o; >> * the hg-cpydev hg extension [7] should be ported to git (optional); >> * the buildbots need to be updated if they are going to pull the >> source from github and use git; >> * the bug tracker will need to be updated to interact with github; >> * the devguide needs to be updated (both to cover the new workflow >> and update links/commands); > > > Yes to all of these (I think; depends if we change how something operates in > terms of associations). > >> >> >> FWIW next weekend (9-10 January) we are organizing a sprint in >> Helsinki, and I'm planning to work on the bug tracker, so I might be >> able to start addressing some of these points. > > > I think deciding how to associate issues with PRs would be a great thing to > work through. Otherwise we need to decide if we want to go with an issue > tracker solution to the NEWS file or if we want a individual file solution > (which happens to be bot-friendly). Those two things are probably the most > critical starting points. > There's also another point that I haven't seen being discussed: how we will handle the author and the committer fields. Currently we only know the committer so, while migrating to git, this can either be duplicated in the author field or just preserved in the committer field leaving the author field empty. Afterwards we can start using the two fields for what they are meant for. A possible problem is how to update them when commits/merges are performed by bots (e.g. if b.p.o automatically converts a patch into a pull request, or if a core dev merges a patch by clicking on a button). Perhaps someone more familiar with Git/GitHub can clarify this (I think it's possible to specify author/committer manually, but I don't know how GitHub handles it on its side). Best Regards, Ezio Melotti > -Brett > >> >> >> Best Regards, >> Ezio Melotti >> >> P.S. enjoy your last few days of vacation while you can ;) >> >> [0]: https://www.python.org/dev/peps/pep-0507/ >> [1]: https://www.python.org/dev/peps/pep-0481/ >> [2]: https://hg.python.org/hooks/file/tip/hgroundup.py >> [3]: https://hg.python.org/hooks/file/tip/hgirker.py >> [4]: https://hg.python.org/hooks/file/tip/hgbuildbot.py >> [5]: https://hg.python.org/hooks/file/tip >> [6]: >> https://hg.python.org/tracker/python-dev/file/tip/extensions/local_replace.py >> [7]: https://bitbucket.org/introom/hg-cpydev From donald at stufft.io Sat Jan 2 17:50:16 2016 From: donald at stufft.io (Donald Stufft) Date: Sat, 2 Jan 2016 17:50:16 -0500 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: <6142851D-D4E3-4AAD-AF8B-CF8580A1EBCF@stufft.io> > On Jan 2, 2016, at 5:35 PM, Ezio Melotti wrote: > > GitHub should also link to b.p.o issues when an issue number appears > in the PR (I don't know if/how this can be done). By default this isn?t allowed, but I think you could use the web hook based support in Github with a bot that has permissions to just inline edit comments so that the typical syntax gets linked to b.p.o. This won?t work in commit messages or such though. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From berker.peksag at gmail.com Sat Jan 2 18:08:49 2016 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Sun, 3 Jan 2016 01:08:49 +0200 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: On Sun, Jan 3, 2016 at 12:35 AM, Ezio Melotti wrote: > On Sat, Jan 2, 2016 at 8:45 PM, Brett Cannon wrote: >> Basically a way to map an issue to a PR (or vice-versa). Probably the >> simplest solution is to allow pasting in a GitHub PR URL or the PR # to make >> the association. The other option is for the bot to accept a command to make >> the association. No reason we can't have both, though. But either way we >> should have a way to connect PRs to issues. >> > > I'll have to think a bit about the best way to implement this and the > UI (I already have some ideas). This can be done without a bot. We just need a webhook server to listen pull_request[1] events. If the action type of the event is "opened" and the pull request title contains "Issue #NNNN", we can communicate with the new Roundup detector. I don't know how the Roundup part works, though. [1] https://developer.github.com/v3/activity/events/types/#pullrequestevent > There's also another point that I haven't seen being discussed: how we > will handle the author and the committer fields. > Currently we only know the committer so, while migrating to git, this > can either be duplicated in the author field or just preserved in the > committer field leaving the author field empty. Afterwards we can > start using the two fields for what they are meant for. > A possible problem is how to update them when commits/merges are > performed by bots (e.g. if b.p.o automatically converts a patch into a > pull request, or if a core dev merges a patch by clicking on a > button). Perhaps someone more familiar with Git/GitHub can clarify > this (I think it's possible to specify author/committer manually, but > I don't know how GitHub handles it on its side). There is no way to store both contributor and committer information via GitHub's web interface. It can be done either manually in terminal or a bot can temporarily set GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL env variables before doing fast-forward merge (we can get the core developer's name and email address from https://developer.github.com/v3/issues/comments/ and https://developer.github.com/v3/users/) From barry at python.org Sat Jan 2 22:41:33 2016 From: barry at python.org (Barry Warsaw) Date: Sat, 2 Jan 2016 22:41:33 -0500 Subject: [core-workflow] We will be moving to GitHub References: Message-ID: <20160102224133.527cda6e@anarchist> I'm personally disappointed of course, and not at all surprised, especially after Guido's stated preference. But I want to be very clear that I accept the decision and want to express my thanks also to Brett, Nick, Donald, Guido, and everyone else who contributed to the conversation. It heartening to know that we all care enough about Python, past, present, and future to want to see it succeed not just technically, but socially as well, and decisions about where and how our code will be developed are at least as much social as they are technical. It's also gratifying to know that a subject that can invoke such passion can for the most part be discussed rationally and civilly within our wonderful community. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Sat Jan 2 22:50:42 2016 From: barry at python.org (Barry Warsaw) Date: Sat, 2 Jan 2016 22:50:42 -0500 Subject: [core-workflow] We will be moving to GitHub References: Message-ID: <20160102225042.629f16d3@anarchist> On Jan 02, 2016, at 06:45 PM, Brett Cannon wrote: >> Some of these things might already be covered by existing PEPs, but I >> don't see them in PEP 507[0] and 481[1] (and I'm getting a bit lost >> among all the competing PEPs and multiple threads across at least a >> couple different MLs :). > >It will be either be a new PEP or the GitHub PEP will simply be rewritten. Please either write a new PEP, or just write some draft new chapters for the developer's guide. Ultimately, anything that a developer has to care about really needs to be described in the devguide. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From nicholas.chammas at gmail.com Sat Jan 2 23:20:45 2016 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Sun, 03 Jan 2016 04:20:45 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: <20160101213045.EB0B6B1408B@webabinitio.net> Message-ID: First-timer here on this list. Just wanted to chime in briefly on a few things. Basically a way to map an issue to a PR (or vice-versa). Probably the simplest solution is to allow pasting in a GitHub PR URL or the PR # to make the association. The other option is for the bot to accept a command to make the association. No reason we can?t have both, though. But either way we should have a way to connect PRs to issues. I have seen many projects ? specifically those using an issue tracker outside of GitHub ? accomplish this by asking their contributors to include the issue number in the title. Two example projects doing this: Apache Spark , Scala Then, those projects would have some kind of bot or hook (like Berker mentioned) use the issue number in the PR title to create a reference over on the issue tracker. For example, here is the script that the Apache Spark project uses to create links on JIRA (their issue tracker) to PRs. And here is an issue where you can see the link that was automatically created to the PR referencing it (under ?Issue Links?). If we end up using a CI service with good GitHub integration like Travis, we may even be able to use it to create the issue links as part of the build?no bot or separate service required. Additionally, if we want all PRs to include an issue reference, for example, that could be automatically enforced. Any PR without an issue reference in the title would fail that check, which shows up as a nice, clear X on the PR, with a link to more detail. I?ll echo what others have said and say that I too am interested in contributing to Python project infrastructure, especially in the area of CI integration and various types of automation that make the lives of contributors and committers easier. I spent a lot of time doing very similar work for the Apache Spark project. I think it?s an important thing to work on; lower friction enables more people to contribute more often, and that?s very important for the long-term health of the project. Nick ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sat Jan 2 23:38:56 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 2 Jan 2016 21:38:56 -0700 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: First, let me add my thanks for sorting this out! On Jan 2, 2016 11:45, "Brett Cannon" wrote: > Well, "support" as in "allow". We won't be keeping Rietveld around (part of this move is so we can get off of Rietveld). I guess I'd missed this point. In my opinion, code review in Github is unpleasant for anything but small PRs and even for those when there's much back-and-forth. At work we switched to Github. We moved code review off to reviewboard a few months later. Setting up the webhooks between the two wasn't hard and code review was a much better experience. Just my 2c. -eric -eric (phone) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.chammas at gmail.com Sat Jan 2 23:40:09 2016 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Sun, 03 Jan 2016 04:40:09 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: <20160102224133.527cda6e@anarchist> References: <20160102224133.527cda6e@anarchist> Message-ID: On Sat, Jan 2, 2016 at 10:41 PM Barry Warsaw barry at python.org wrote: It heartening to know that we all care enough about Python, past, present, and future to want to see it succeed not just technically, but socially as well, and decisions about where and how our code will be developed are at least as much social as they are technical. It?s also gratifying to know that a subject that can invoke such passion can for the most part be discussed rationally and civilly within our wonderful community. I?d like to echo this sentiment as well. I?ll also add that Python now joins an impressive list of programming languages hosted and developed on GitHub . Some of those languages just use GitHub?s issue tracker (e.g. Go ), others just use GitHub pull requests (e.g. Swift , Ruby ), and others use the full feature set (e.g. Rust ). I hope this transition both enables more people to get involved with Python and lessens the contribution burden on committers. Cheers! Nick ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Jan 3 00:08:20 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 2 Jan 2016 22:08:20 -0700 Subject: [core-workflow] Longer term idea: consider Rust's homu for CPython merge gating In-Reply-To: References: Message-ID: On Thu, Dec 31, 2015 at 1:21 PM, Brett Cannon wrote: > Thanks for the link! For those that didn't look at the link, Homu is > actually written in Python and not Rust (the website for the project is > what's written in Rust). > > Berker has actually said he is beginning to look into writing a bot for > our needs so he might be interested in this. > > A quick perusal suggests it probably needs to be modified to understand > supporting multiple branches. I also don't know if it supports the > fast-forwarding/rebasing commits we want, but I doubt that would be > difficult to add. It probably also needs to grow an extension system > somehow to support custom NEWS entries for us. > > But it probably would be really nice for the Python and Rust communities > to share some of the infrastructure burden by working on the same tooling > (especially if we even consider using their auto-assignment to help get > patches reviewed). > While that's true, the ages of the projects are so different that I wonder if our workflows have that much in common. And it seems there are quite a few tools that work like this. In general I like the idea of a commit queue; we built one at Dropbox which has been very useful in keeping our master green. Based on this experience, I note that testing each commit before it's merged limits the rate at which commits can be merged, unless you play some tricks that occasionally backfire. Taking homu's brief description literally, it would seem that if it takes e.g. 10 minutes to run the tests you can't accept more than 6 commits per hour. I don't recall how long Python's tests take, but I I wouldn't be surprised if it was a lot longer than 10 minutes. Now, you can reduce the running time by parallelizing test runs, and Python doesn't see that many commits typically, so perhaps this isn't a problem. But for larger projects it may be -- it definitely was a big concern at Dropbox, and I recall hearing about some astronomical stats from the Chromium project. At Dropbox we solved this by not literally testing each commit after the last one has been merged. Instead, for each commit to be tested, we create a throwaway branch where we merge the commit with the current HEAD on the master branch, test the result, and then (if the tests pass) re-merge the commit onto master. This way we can test many commits in parallel (and yes, we pay a lot of money to Amazon for the needed capacity :-). It is *possible* that this strategy introduces a failure in master, but pretty unlikely if the final merge succeeds without conflicts (on a git merge conflict we start over). We re-test after the commit has landed on master so we will still detect such failures -- but in practice there are some other reasons why the post-merge test can fail (typically due to flaky tests) and those are much more common than "commit races" (as we've dubbed the phenomenon of two commits that each individually pass the tests, and merge without conflicts, yet introduce a test failure when combined). FWIW the tools we use at Dropbox also linearize commit history. We never keep the micro-commits used by the developer during (or before) the code review phase -- that history is saved forever in our separate code review tool. The CI tool runs tests at each code review iteration, but without the "merge with latest HEAD" step described above -- that's only done after the developer and reviewer have reached agreement on landing the commit (or when the developer explicitly rebases their short-term development branch). --Guido > On Wed, 30 Dec 2015 at 20:21 Nick Coghlan wrote: > >> (Note: near term, this probably isn't a useful idea, as the current >> GitHub & GitLab proposals aren't proposing the introduction of merge >> gating, merely advisory testing of PRs, so folks will remain free to >> merge patches locally and push them up to the master repository. >> However, I think pre-merge testing is a good idea long term, with >> approving our own PRs taking the place of pushing directly to the >> development and maintenance branches) >> >> I recently came across an old article [1] by Rust's Graydon Hoare >> regarding their approach to pre-merge testing of commits using GitHub >> PRs and Buildbot. Following up on that brought me to a more up to date >> explanation of their current bot setup [2], and the http://homu.io/ >> service for running it against a project without hosting your own >> instance (in our case, if we did run a service like Homu, we'd likely >> still want to host our own instance, as Mozilla do for Rust and >> Servo). >> >> Similar to OpenStack's Zuul, Homu serialises commits, ensuring that >> the test suite is passing *before* code lands on the target branch. >> The relevant differences are that Homu is designed to use GitHub PRs >> rather than Gerrit reviews as the event source, and Travis CI and >> Buildbot rather than Jenkins as the CI backends for actually running >> the integration tests. (Homu is also currently missing Zuul's >> "speculative test pipeline" feature, which allows it to test multiple >> commits in parallel on the assumption that most of them will pass due >> to prior testing in isolation, but has a separate feature that allows >> PRs to be tagged for inclusion in "rollup" commits that land several >> changes at once after testing them as a group) >> >> The current merge queue for Rust itself can be seen at >> http://buildbot.rust-lang.org/homu/queue/rust >> >> Regards, >> Nick. >> >> [1] http://graydon2.dreamwidth.org/1597.html >> [2] >> http://huonw.github.io/blog/2015/03/rust-infrastructure-can-be-your-infrastructure/ >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> core-workflow mailing list >> core-workflow at python.org >> https://mail.python.org/mailman/listinfo/core-workflow >> This list is governed by the PSF Code of Conduct: >> https://www.python.org/psf/codeofconduct >> > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 3 00:40:23 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Jan 2016 15:40:23 +1000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: <20160102225042.629f16d3@anarchist> References: <20160102225042.629f16d3@anarchist> Message-ID: On 3 January 2016 at 13:50, Barry Warsaw wrote: > On Jan 02, 2016, at 06:45 PM, Brett Cannon wrote: > >>> Some of these things might already be covered by existing PEPs, but I >>> don't see them in PEP 507[0] and 481[1] (and I'm getting a bit lost >>> among all the competing PEPs and multiple threads across at least a >>> couple different MLs :). >> >>It will be either be a new PEP or the GitHub PEP will simply be rewritten. > > Please either write a new PEP, or just write some draft new chapters for the > developer's guide. Ultimately, anything that a developer has to care about > really needs to be described in the devguide. For the last hosting transition, PEP 374 covered the process of choosing a distributed VCS, while PEP 385 covered the actual Subversion -> Mercurial transition, and I think that's a good way to go. In particular, a separate "Migrating from hg.python.org to GitHub" PEP provides a place to document the TODO list for the transition, and the outcomes of any smaller decisions that will need to be made as an incidental part of the migration (like handling Author/Committer data for old commits), which aren't generally relevant to future contributors (so don't belong in the developer guide), but also weren't part of the repository hosting decision making process (so don't really belong in PEP 507). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Sun Jan 3 01:01:29 2016 From: brett at python.org (Brett Cannon) Date: Sun, 03 Jan 2016 06:01:29 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: <20160102225042.629f16d3@anarchist> Message-ID: I'm planning to write a separate PEP. I'll start outlining what I think we need to accomplish and in what order on Monday and that will be the kick-off for the outline of the PEP so we can get a good feel for the work to be done (roughly). On Sat, 2 Jan 2016, 21:40 Nick Coghlan wrote: > On 3 January 2016 at 13:50, Barry Warsaw wrote: > > On Jan 02, 2016, at 06:45 PM, Brett Cannon wrote: > > > >>> Some of these things might already be covered by existing PEPs, but I > >>> don't see them in PEP 507[0] and 481[1] (and I'm getting a bit lost > >>> among all the competing PEPs and multiple threads across at least a > >>> couple different MLs :). > >> > >>It will be either be a new PEP or the GitHub PEP will simply be > rewritten. > > > > Please either write a new PEP, or just write some draft new chapters for > the > > developer's guide. Ultimately, anything that a developer has to care > about > > really needs to be described in the devguide. > > For the last hosting transition, PEP 374 covered the process of > choosing a distributed VCS, while PEP 385 covered the actual > Subversion -> Mercurial transition, and I think that's a good way to > go. > > In particular, a separate "Migrating from hg.python.org to GitHub" PEP > provides a place to document the TODO list for the transition, and the > outcomes of any smaller decisions that will need to be made as an > incidental part of the migration (like handling Author/Committer data > for old commits), which aren't generally relevant to future > contributors (so don't belong in the developer guide), but also > weren't part of the repository hosting decision making process (so > don't really belong in PEP 507). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 3 01:06:23 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Jan 2016 16:06:23 +1000 Subject: [core-workflow] Longer term idea: consider Rust's homu for CPython merge gating In-Reply-To: References: Message-ID: On 3 January 2016 at 15:08, Guido van Rossum wrote: > In general I like the idea of a commit queue; we built one at Dropbox which > has been very useful in keeping our master green. Based on this experience, > I note that testing each commit before it's merged limits the rate at which > commits can be merged, unless you play some tricks that occasionally > backfire. Taking homu's brief description literally, it would seem that if > it takes e.g. 10 minutes to run the tests you can't accept more than 6 > commits per hour. I don't recall how long Python's tests take, but I I > wouldn't be surprised if it was a lot longer than 10 minutes. It's less than 10 minutes on a modern laptop (although I admit I haven't done a -uall run in a while), but longer than that on the Buildbot fleet (since some of the buildbots aren't particularly fast). > Now, you can reduce the running time by parallelizing test runs, and Python > doesn't see that many commits typically, so perhaps this isn't a problem. > But for larger projects it may be -- it definitely was a big concern at > Dropbox, and I recall hearing about some astronomical stats from the > Chromium project. > > At Dropbox we solved this by not literally testing each commit after the > last one has been merged. Instead, for each commit to be tested, we create a > throwaway branch where we merge the commit with the current HEAD on the > master branch, test the result, and then (if the tests pass) re-merge the > commit onto master. This way we can test many commits in parallel (and yes, > we pay a lot of money to Amazon for the needed capacity :-). As far I'm aware, OpenStack's Zuul mergebot is currently best in class at solving this problem: * while a patch is in review, they run a "check" test against master to see if the change works at all * to be approved for merge, patches need both a clean check run and approval from human reviewers * once patches are approved for merge, they go into a merge queue, where each patch is applied atop the previous one * Zuul runs up to N test runs in parallel, merging them in queue order as they finish successfully * if any test run fails, any subsequent test runs (finished or not) are discarded, the offending patch is removed from the merge queue, and Zuul rebuilds the queue with the failing patch removed It's a lot like a CPU pipeline in that regard - when everything goes well, patches are merged as fast as they're approved, just with a fixed time delay corresponding to the length of time it takes to run the test suite. If one of the merges fails, then it's like branch prediction in a CPU failing - you have to flush the pipeline and start over again from the point of the failure. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Sun Jan 3 09:43:20 2016 From: barry at python.org (Barry Warsaw) Date: Sun, 3 Jan 2016 09:43:20 -0500 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: <20160102225042.629f16d3@anarchist> Message-ID: <20160103094320.1f03d12b@limelight.wooz.org> On Jan 03, 2016, at 03:40 PM, Nick Coghlan wrote: >For the last hosting transition, PEP 374 covered the process of >choosing a distributed VCS, while PEP 385 covered the actual >Subversion -> Mercurial transition, and I think that's a good way to >go. Yes, for the transition tasks, a PEP makes sense. This is essentially a one-off document, at least for the next 5 or 6 years . But I don't think a PEP is the right place to describe the things that the average developer (core or otherwise) needs to know on a daily basis. That goes in the devguide. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From brett at python.org Sun Jan 3 12:35:20 2016 From: brett at python.org (Brett Cannon) Date: Sun, 03 Jan 2016 17:35:20 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: On Sat, 2 Jan 2016 at 20:39 Eric Snow wrote: > First, let me add my thanks for sorting this out! > > On Jan 2, 2016 11:45, "Brett Cannon" wrote: > > Well, "support" as in "allow". We won't be keeping Rietveld around (part > of this move is so we can get off of Rietveld). > > I guess I'd missed this point. In my opinion, code review in Github is > unpleasant for anything but small PRs and even for those when there's much > back-and-forth. At work we switched to Github. We moved code review off > to reviewboard a few months later. Setting up the webhooks between the two > wasn't hard and code review was a much better experience. Just my 2c. > No one proposed that workflow so it wasn't considered (and I'm obviously not about to start the process again ;). If we find that GitHub isn't working out for code review then we can discuss how to remedy it, but that's not something to consider until we have been done with the transition for several months at least for people to form an informed opinion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skrah.temporarily at gmail.com Sun Jan 3 13:28:40 2016 From: skrah.temporarily at gmail.com (Stefan Krah) Date: Sun, 3 Jan 2016 18:28:40 +0000 (UTC) Subject: [core-workflow] We will be moving to GitHub References: Message-ID: Eric Snow writes: > I guess I'd missed this point.? In my opinion, code review in Github is unpleasant for anything but small PRs and even for those when there's much back-and-forth.? At work we switched to Github.? We moved code review off to reviewboard a few months later.? Setting up the webhooks between the two wasn't hard and code review was a much better experience.? Just my 2c. Agreed. Our current Rietveld setup is superior and much less distracting. Like the Rietveld house vs. Victorian architecture. Stefan Krah From brett at python.org Sun Jan 3 14:26:54 2016 From: brett at python.org (Brett Cannon) Date: Sun, 03 Jan 2016 19:26:54 +0000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: Rietveld is no longer an option as our fork of the project is unmaintained (it was one of the key reasons we even started this process). On Sun, Jan 3, 2016, 10:28 Stefan Krah wrote: > Eric Snow writes: > > I guess I'd missed this point. In my opinion, code review in Github is > unpleasant for anything but small PRs and even for those when there's much > back-and-forth. At work we switched to Github. We moved code review off > to > reviewboard a few months later. Setting up the webhooks between the two > wasn't hard and code review was a much better experience. Just my 2c. > > Agreed. Our current Rietveld setup is superior and much less distracting. > > Like the Rietveld house vs. Victorian architecture. > > > Stefan Krah > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Jan 3 14:32:07 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 3 Jan 2016 11:32:07 -0800 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: (Though honestly if we were okay with hosting by Google, Rietveld would still be an option. But I agree we should first figure out whether we can live with GitHub's review.) On Sun, Jan 3, 2016 at 11:26 AM, Brett Cannon wrote: > Rietveld is no longer an option as our fork of the project is unmaintained > (it was one of the key reasons we even started this process). > > On Sun, Jan 3, 2016, 10:28 Stefan Krah > wrote: > >> Eric Snow writes: >> > I guess I'd missed this point. In my opinion, code review in Github is >> unpleasant for anything but small PRs and even for those when there's much >> back-and-forth. At work we switched to Github. We moved code review off >> to >> reviewboard a few months later. Setting up the webhooks between the two >> wasn't hard and code review was a much better experience. Just my 2c. >> >> Agreed. Our current Rietveld setup is superior and much less distracting. >> >> Like the Rietveld house vs. Victorian architecture. >> >> >> Stefan Krah >> _______________________________________________ >> core-workflow mailing list >> core-workflow at python.org >> https://mail.python.org/mailman/listinfo/core-workflow >> This list is governed by the PSF Code of Conduct: >> https://www.python.org/psf/codeofconduct > > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Jan 3 16:23:22 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 3 Jan 2016 13:23:22 -0800 Subject: [core-workflow] Longer term idea: consider Rust's homu for CPython merge gating In-Reply-To: References: Message-ID: Ah, the last part there is a nice algorithm. It determines the intended merge order when patches are submitted to the queue, then runs tests for multiple patches in parallel. So, assuming most patches pass their tests, the rate at which patches can land is limited only by how fast it can run parallelized tests, and as long as the queue isn't congested, each patch lands roughly as soon as its tests succeed. On Sat, Jan 2, 2016 at 10:06 PM, Nick Coghlan wrote: > On 3 January 2016 at 15:08, Guido van Rossum wrote: > > In general I like the idea of a commit queue; we built one at Dropbox > which > > has been very useful in keeping our master green. Based on this > experience, > > I note that testing each commit before it's merged limits the rate at > which > > commits can be merged, unless you play some tricks that occasionally > > backfire. Taking homu's brief description literally, it would seem that > if > > it takes e.g. 10 minutes to run the tests you can't accept more than 6 > > commits per hour. I don't recall how long Python's tests take, but I I > > wouldn't be surprised if it was a lot longer than 10 minutes. > > It's less than 10 minutes on a modern laptop (although I admit I > haven't done a -uall run in a while), but longer than that on the > Buildbot fleet (since some of the buildbots aren't particularly fast). > > > Now, you can reduce the running time by parallelizing test runs, and > Python > > doesn't see that many commits typically, so perhaps this isn't a problem. > > But for larger projects it may be -- it definitely was a big concern at > > Dropbox, and I recall hearing about some astronomical stats from the > > Chromium project. > > > > At Dropbox we solved this by not literally testing each commit after the > > last one has been merged. Instead, for each commit to be tested, we > create a > > throwaway branch where we merge the commit with the current HEAD on the > > master branch, test the result, and then (if the tests pass) re-merge the > > commit onto master. This way we can test many commits in parallel (and > yes, > > we pay a lot of money to Amazon for the needed capacity :-). > > As far I'm aware, OpenStack's Zuul mergebot is currently best in class > at solving this problem: > > * while a patch is in review, they run a "check" test against master > to see if the change works at all > * to be approved for merge, patches need both a clean check run and > approval from human reviewers > * once patches are approved for merge, they go into a merge queue, > where each patch is applied atop the previous one > * Zuul runs up to N test runs in parallel, merging them in queue order > as they finish successfully > * if any test run fails, any subsequent test runs (finished or not) > are discarded, the offending patch is removed from the merge queue, > and Zuul rebuilds the queue with the failing patch removed > > It's a lot like a CPU pipeline in that regard - when everything goes > well, patches are merged as fast as they're approved, just with a > fixed time delay corresponding to the length of time it takes to run > the test suite. If one of the merges fails, then it's like branch > prediction in a CPU failing - you have to flush the pipeline and start > over again from the point of the failure. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 3 19:35:38 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Jan 2016 10:35:38 +1000 Subject: [core-workflow] We will be moving to GitHub In-Reply-To: References: Message-ID: On 4 January 2016 at 03:35, Brett Cannon wrote: > > On Sat, 2 Jan 2016 at 20:39 Eric Snow wrote: >> >> First, let me add my thanks for sorting this out! >> >> On Jan 2, 2016 11:45, "Brett Cannon" wrote: >> > Well, "support" as in "allow". We won't be keeping Rietveld around (part >> > of this move is so we can get off of Rietveld). >> >> I guess I'd missed this point. In my opinion, code review in Github is >> unpleasant for anything but small PRs and even for those when there's much >> back-and-forth. At work we switched to Github. We moved code review off to >> reviewboard a few months later. Setting up the webhooks between the two >> wasn't hard and code review was a much better experience. Just my 2c. > > No one proposed that workflow so it wasn't considered (and I'm obviously not > about to start the process again ;). If we find that GitHub isn't working > out for code review then we can discuss how to remedy it, but that's not > something to consider until we have been done with the transition for > several months at least for people to form an informed opinion. The useful aspect of the GitHub-as-platform business model in this case is that while PRs are the *default* review workflow, GitHub's APIs are deliberately designed to let people slot in their own review tools. pip's repo shows what the PR interface looks like with Reviewable enabled, for instance: https://github.com/pypa/pip/pull/3338 (there's just an extra button to jump to the review in the external review tool) For the OpenStack workflow fans, there's http://gerrithub.io/ I can't find any examples of direct integration of Rietveld with GitHub, so that would presumably require setting up some webhooks, similar to what Eric described doing for Reviewboard. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Jan 3 19:57:20 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Jan 2016 10:57:20 +1000 Subject: [core-workflow] Longer term idea: consider Rust's homu for CPython merge gating In-Reply-To: References: Message-ID: On 4 January 2016 at 07:23, Guido van Rossum wrote: > Ah, the last part there is a nice algorithm. It determines the intended > merge order when patches are submitted to the queue, then runs tests for > multiple patches in parallel. So, assuming most patches pass their tests, > the rate at which patches can land is limited only by how fast it can run > parallelized tests, and as long as the queue isn't congested, each patch > lands roughly as soon as its tests succeed. Yeah, I first saw a talk on it at linux.conf.au a couple of years ago, and was really impressed. They have a good write-up on the system here: http://docs.openstack.org/infra/zuul/gating.html For CPython though, OpenHub indicates our commit rate is on the order of 15 per day (5754 commits in the last 12 months), so even if GitHub gets us a nice uptick in landing submitted patches, we still have a lot of headroom before the main bottleneck becomes anything other than availability of reviewers. Cheers, Nick. P.S. While I think it would be overkill for CPython, folks working with distributed systems may also be interested in a neat system called ElasticRecheck that OpenStack use to identify intermittent failures based on patterns detected in log files: http://docs.openstack.org/infra/elastic-recheck/readme.html#idea Relevant dashboard for the OpenStack integrated release gate: http://status.openstack.org/elastic-recheck/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Mon Jan 4 19:42:27 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 00:42:27 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition Message-ID: So consider this the starting discussion of the PEP that will be the hg.python.org -> GitHub transition PEP that I will be in charge of. Once we have general agreement on the steps necessary I will start the actual PEP and check it in, but I figure there's no point in have a skeleton PEP if we can't agree on the skeleton. :) While I list steps influencing all the repos, I want to focus on the ones stopping any repo from moving over for now, expanding what we worry about to the cpython repo as we knock blockers down until we move everything over and start adding GitHub perks. The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, devguide, and cpython. I also think that's essentially the order we should migrate them over. Some things will need to be universally handled before we transition a single repo, while other things are only a blocker for some of the repos. Universal blockers ============== There are four blockers that must be resolved before we even consider moving a repo over. They can be solved in parallel, but they all need to have a selected solution before we can move even the devinabox repo. First, we need to decide how we are going to handle adding all the core devs to GitHub. Are we simply going to add all of them to the python organization, or do we want something like a specific python-dev gteamroup that gets added to all of the relevant repos? Basically I'm not sure how picky we want to be about the people who already have organization access on GitHub about them implicitly getting access to the cpython repo at the end of the day (and the same goes for any of the other repos in the python organization). For tracking names, I figure we will create a file in the devguide where people can check in their GitHub usernames and I can manually add people as people add themselves to the file. Second, CLA enforcement. As of right now people go to https://www.python.org/psf/contrib/contrib-form/, fill in the form, and then Ewa gets an email where she manually flips a flag in Roundup. If we want to use a web hook to verify someone has signed a CLA then we need to decide where the ground truth for CLAs are. Do we want to keep using Roundup to manage CLA agreements and thus add a GitHub field in bugs.python.org for people's profile and a web hook or bot that will signal if someone has the flag flipped on bugs.python.org? Or is there some prepackaged service that we can use that will keep track of this which would cause us to not use Roundup (which might be easier, but depending on the service require everyone to re-sign)? There's also the issue of supporting people who want to submit code by uploading a patch to bugs.python.org but not use GitHub. Either way I don't want to have to ask everyone who submits a PR what their bugs.python.org username is and then go check that manually. Third, how do we want to do the repo conversions? We need to choose the tool(s) and command(s) that we want to use. There was mention of wanting a mapping from hg commit ID to git commit ID. If we have that we could have a static bugs.python.org/commit/ page that had the mapping embedded in some JavaScript and if matched then we just forward them to the corresponding GitHub commit page, otherwise just blindly forward to GitHub and assume the ID is git-only, giving us a stable URL for commit web views. Fourth, for the ancillary repos of devinabox, peps, benchmarks, and devguide, do we care if we use the GitHub merge button for PRs or do we want to enforce a linear history with all repos? We just need to decide if care about linear histories and then we can move forward since any bot we create won't block us from using GitHub. Those four things are enough to move devinabox over. It probably is enough for the benchmarks suite, but I have an email to speed@ asking if people want to use this opportunity to re-evaluate the benchmark suite and make any changes that will affect repo size (e.g., use pip to pull in the libraries and frameworks used by a benchmark rather than vendoring their code, making the repo much smaller). Website-related stuff ================ This also almost gets us the peps repo, but we do need to figure out how to change the website to build from the git checkout rather than an hg one. Same goes for the devguide. It would be great if we can set up web hooks to immediately trigger rebuilds of those portions of the sites instead of having to wait until a cronjob triggers. CPython requirements ================= There are six things to work out before we move over cpython. First, do we want to split out Python 2 branches into their own repo? There might be a clone size benefit which obviously is nice for people on slow Internet connections. It also clearly separates out Python 2 from 3 and lets those who prefer to focus on one compared to the other do that more easily. It does potentially make any single fix that spans 2 and 3 require a bit more work since it won't be an intra-clone change. We could also contemplate sub-repos for things like the Doc/ or Tools/ directories (although I doubt it's worth it). Second, do we want all fixes to go into master and then cherry-pick into other branches, or do we want to follow our current practice of going into the active feature branch and then merge into master? I personally prefer the former and I think most everyone else does as well, but I thought it should be at least thought about. Third, how to handle Misc/NEWS? We can add a NEWS field to bugs.python.org and then generate the NEWS file by what issues are tied to what version and when they were closed. The other approach is to create a file per NEWS entry in a version-specific directory (Larry created code for hg already for this to give people an idea: http://bugs.python.org/issue18967). Then when we cut a release we run a tool the slurps up all of the relevant files -- which includes files in the directory for the next feature release which represent fixes which were cherry picked -- and generates the NEWS file for the final release. The per-file approach is bot-friendly and also CLI-friendly, but potentially requires more tooling and I don't know if people feel news entries should be tied to the issue or in the repo (although that assumes people find tweaking Roundup easy :). Fourth, we need to decide exactly what commands we expect core devs to run initially for committing code. Since we agreed to a linear history we need to document exactly what we expect people to do for a PR to get it into their git repo. This will go into the devguide -- probably will want to start a github branch at some point -- and will act as the commands the bot will want to work off of. Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we flatten them, I believe the PR creators will get credit in the commit as the author while the core dev committing will be flagged as the person doing the merge (someone correct me if I'm wrong because if I am this whole point is silly). With the commits containing credit directly, we can either automatically generate Misc/ACKS like the NEWS file or simply drop it for future contributors and just leave the file for past contributors since git will have kept track for us. Six, we will need to update our Buildbot fleet. This gets us to the bare minimum needed to function. Parity with hg.python.org ---------------------------------- For parity, there are some Roundup integrations that will be necessary, like auto-generating links, posting commits to #python-dev on IRC, etc. I'm not sure if people want to block until that is all in place or not. I do think we should make sure there is some web hook that can take an issue # from the title of a PR and automatically posts to the corresponding issue on bugs.python.org that the PR exists. If people disagree then feel free to say so. Adding perks ========== Now we get to some added stuff that we never had on our own infrastructure. :) We should wire up CI for all PRs. I don't know if we want to go with Travis, Codeship, or what CI provider, but we should definitely hook it up and fully utilize the resource. This could even include running doctest over the docs, making sure the reST markup is accurate, etc. Do we need to set up a web hook to trigger website rebuilds? We should at least have a mirror on Read the Docs that is triggered by web hook so that we have a backup of the documentation (if possible; not sure how custom our Sphinx setup is compared to what they require to work). We should try to get test coverage wired up as well per CI. I don't know if coveralls.io or some other provider is best, but we should see what is available and find out if we can use them to either get basic coverage or thorough coverage (read https://hg.python.org/devinabox/file/tip/README#l124 to see what thorough coverage entails, but it does require a checkout of coverage.py). We should build a bot. It must use a Monty Python reference to trigger (black-knight, none-shall-pass, man-from-scene-24, five-questions, what-is-your-quest, what-is-your-favourite-colour, etc.; obviously I'm leaning towards the Black Knight or Bridge of Death scenes from the Holy Grail for inspiration since they deal with blocking you from doing something). It should handle specifying the commit message, what branches to commit to/cherry pick into, and a NEWS entry (if necessary). I don't know if it needs to do anything else as a requirement. It should probably implement a commit queue like Zuul or Homu (and both of those can be considered as the basis of the bot). Also gating commits on passing a test run probably would also be good. I'm sure we will want to use some labels and milestones to track what PRs are for what versions, if they are blocked on something, etc. --- Long email! :) I think that is my current brain dumped in email form. As I said at the beginning, I think we should focus on what is blocking the easiest repos first and then just keep knocking down blockers as we try to move over more repos. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jan 4 20:08:19 2016 From: donald at stufft.io (Donald Stufft) Date: Mon, 4 Jan 2016 20:08:19 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: <93574A50-8E31-49D1-80A4-590AC4C347D7@stufft.io> > On Jan 4, 2016, at 7:42 PM, Brett Cannon wrote: > > So consider this the starting discussion of the PEP that will be the hg.python.org -> GitHub transition PEP that I will be in charge of. Once we have general agreement on the steps necessary I will start the actual PEP and check it in, but I figure there's no point in have a skeleton PEP if we can't agree on the skeleton. :) While I list steps influencing all the repos, I want to focus on the ones stopping any repo from moving over for now, expanding what we worry about to the cpython repo as we knock blockers down until we move everything over and start adding GitHub perks. > > The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, devguide, and cpython. I also think that's essentially the order we should migrate them over. Some things will need to be universally handled before we transition a single repo, while other things are only a blocker for some of the repos. > > Universal blockers > ============== > There are four blockers that must be resolved before we even consider moving a repo over. They can be solved in parallel, but they all need to have a selected solution before we can move even the devinabox repo. > > First, we need to decide how we are going to handle adding all the core devs to GitHub. Are we simply going to add all of them to the python organization, or do we want something like a specific python-dev gteamroup that gets added to all of the relevant repos? Basically I'm not sure how picky we want to be about the people who already have organization access on GitHub about them implicitly getting access to the cpython repo at the end of the day (and the same goes for any of the other repos in the python organization). For tracking names, I figure we will create a file in the devguide where people can check in their GitHub usernames and I can manually add people as people add themselves to the file. We should have a team for each logical group of access I think. We can add people to multiple teams, or to just one team. The list of people who can access *all* repos on the Python org is https://caremad.io/s/7Q0wcgOcoE/, everyone else (currently) has restricted access to one or more repositories. > > Second, CLA enforcement. As of right now people go to https://www.python.org/psf/contrib/contrib-form/ , fill in the form, and then Ewa gets an email where she manually flips a flag in Roundup. If we want to use a web hook to verify someone has signed a CLA then we need to decide where the ground truth for CLAs are. Do we want to keep using Roundup to manage CLA agreements and thus add a GitHub field in bugs.python.org for people's profile and a web hook or bot that will signal if someone has the flag flipped on bugs.python.org ? Or is there some prepackaged service that we can use that will keep track of this which would cause us to not use Roundup (which might be easier, but depending on the service require everyone to re-sign)? There's also the issue of supporting people who want to submit code by uploading a patch to bugs.python.org but not use GitHub. Either way I don't want to have to ask everyone who submits a PR what their bugs.python.org username is and then go check that manually. There is CLAHub (https://www.clahub.com/ ) but I don?t have any idea how good it is, I just know of it?s existence. > > Third, how do we want to do the repo conversions? We need to choose the tool(s) and command(s) that we want to use. There was mention of wanting a mapping from hg commit ID to git commit ID. If we have that we could have a static bugs.python.org/commit/ page that had the mapping embedded in some JavaScript and if matched then we just forward them to the corresponding GitHub commit page, otherwise just blindly forward to GitHub and assume the ID is git-only, giving us a stable URL for commit web views. This is the tool I used for the demo repo, it seemed to work ok as long as I ran it on Linux: https://github.com/frej/fast-export It lets you annotate the git commits with the HG hash as a ?git note?, though tool support for git notes doesn?t seem to be very good. Github doesn?t display them but they are available in the CLI if you run the right command to ask for the hg hash that a particular git commit hash came from. > > Fourth, for the ancillary repos of devinabox, peps, benchmarks, and devguide, do we care if we use the GitHub merge button for PRs or do we want to enforce a linear history with all repos? We just need to decide if care about linear histories and then we can move forward since any bot we create won't block us from using GitHub. Personally for most repositories I would just use the GitHub merge button. > > Those four things are enough to move devinabox over. It probably is enough for the benchmarks suite, but I have an email to speed@ asking if people want to use this opportunity to re-evaluate the benchmark suite and make any changes that will affect repo size (e.g., use pip to pull in the libraries and frameworks used by a benchmark rather than vendoring their code, making the repo much smaller). > > Website-related stuff > ================ > This also almost gets us the peps repo, but we do need to figure out how to change the website to build from the git checkout rather than an hg one. Same goes for the devguide. It would be great if we can set up web hooks to immediately trigger rebuilds of those portions of the sites instead of having to wait until a cronjob triggers. We don?t *actually* need to do much here. We could have a cronjob (or webhook based daemon or something) that used hg-git to do ``hg pull`` from GitHub onto hg.python.org (and probably a mirror on git.python.org too). That would A) allow read only tooling that is currently pointing at hg.python.org to continue to work unmodified and B) allow people to interact with our repos, in a read only fashion, without ever talking to GitHub. Couple that with the ability to still upload patches to the bug tracker and people can still contribute without ever personally sending a packet of data to Github. Of course it?d be nice to get the website itself pulling straight from Github (possible also using a web hook based daemon) though it could also either use hg-git, or just switch to Git. Either way should work. > > CPython requirements > ================= > There are six things to work out before we move over cpython. First, do we want to split out Python 2 branches into their own repo? There might be a clone size benefit which obviously is nice for people on slow Internet connections. It also clearly separates out Python 2 from 3 and lets those who prefer to focus on one compared to the other do that more easily. It does potentially make any single fix that spans 2 and 3 require a bit more work since it won't be an intra-clone change. We could also contemplate sub-repos for things like the Doc/ or Tools/ directories (although I doubt it's worth it). Personally I feel like we should just have all of the branches live in the same repository. I don?t think there?s going to be much to gain by stripping out the other branches and I think that the downside of trying to work on 2+ repositories is a hefty price to pay. A fresh clone on the demo repo I setup has a .git of 156M. > > Second, do we want all fixes to go into master and then cherry-pick into other branches, or do we want to follow our current practice of going into the active feature branch and then merge into master? I personally prefer the former and I think most everyone else does as well, but I thought it should be at least thought about. I think it will work best if fixes go into master. I find less problems with people writing patches/PRs against the wrong branch that way. > > Third, how to handle Misc/NEWS? We can add a NEWS field to bugs.python.org and then generate the NEWS file by what issues are tied to what version and when they were closed. The other approach is to create a file per NEWS entry in a version-specific directory (Larry created code for hg already for this to give people an idea: http://bugs.python.org/issue18967 ). Then when we cut a release we run a tool the slurps up all of the relevant files -- which includes files in the directory for the next feature release which represent fixes which were cherry picked -- and generates the NEWS file for the final release. The per-file approach is bot-friendly and also CLI-friendly, but potentially requires more tooling and I don't know if people feel news entries should be tied to the issue or in the repo (although that assumes people find tweaking Roundup easy :). I haven?t actually used it yet, but a friend has recently made https://pypi.python.org/pypi/towncrier which might be useful for this. > > Fourth, we need to decide exactly what commands we expect core devs to run initially for committing code. Since we agreed to a linear history we need to document exactly what we expect people to do for a PR to get it into their git repo. This will go into the devguide -- probably will want to start a github branch at some point -- and will act as the commands the bot will want to work off of. In case folks don?t know, github makes the PRs available as a remote HEAD that you can check out directly from a clone, might be useful for this. > > Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we flatten them, I believe the PR creators will get credit in the commit as the author while the core dev committing will be flagged as the person doing the merge (someone correct me if I'm wrong because if I am this whole point is silly). With the commits containing credit directly, we can either automatically generate Misc/ACKS like the NEWS file or simply drop it for future contributors and just leave the file for past contributors since git will have kept track for us. Git allows you to do it either way, by default it tracks the author and the committer separately so people will get credit, but if someone is just apply a diff then that obviously won?t happen by default (but people can still use the relevant options to make it happen). > > Six, we will need to update our Buildbot fleet. > > This gets us to the bare minimum needed to function. > > Parity with hg.python.org > ---------------------------------- > For parity, there are some Roundup integrations that will be necessary, like auto-generating links, posting commits to #python-dev on IRC, etc. I'm not sure if people want to block until that is all in place or not. I do think we should make sure there is some web hook that can take an issue # from the title of a PR and automatically posts to the corresponding issue on bugs.python.org that the PR exists. If people disagree then feel free to say so. Github has a built in IRC bot for commits FWIW. I agree with the issue # bit. > > Adding perks > ========== > Now we get to some added stuff that we never had on our own infrastructure. :) > > We should wire up CI for all PRs. I don't know if we want to go with Travis, Codeship, or what CI provider, but we should definitely hook it up and fully utilize the resource. This could even include running doctest over the docs, making sure the reST markup is accurate, etc. I like Travis a lot and I know the folks behind it, I?m sure they?d be happy to help. > > Do we need to set up a web hook to trigger website rebuilds? We should at least have a mirror on Read the Docs that is triggered by web hook so that we have a backup of the documentation (if possible; not sure how custom our Sphinx setup is compared to what they require to work). I?m sure Eric would be willing to help to make this happen. > > We should try to get test coverage wired up as well per CI. I don't know if coveralls.io or some other provider is best, but we should see what is available and find out if we can use them to either get basic coverage or thorough coverage (read https://hg.python.org/devinabox/file/tip/README#l124 to see what thorough coverage entails, but it does require a checkout of coverage.py). I prefer codecov, but it shouldn?t be too hard to do. I tried to get Python + C coverage checking in the demo with that, but I failed at making the C coverage work. > > We should build a bot. It must use a Monty Python reference to trigger (black-knight, none-shall-pass, man-from-scene-24, five-questions, what-is-your-quest, what-is-your-favourite-colour, etc.; obviously I'm leaning towards the Black Knight or Bridge of Death scenes from the Holy Grail for inspiration since they deal with blocking you from doing something). It should handle specifying the commit message, what branches to commit to/cherry pick into, and a NEWS entry (if necessary). I don't know if it needs to do anything else as a requirement. It should probably implement a commit queue like Zuul or Homu (and both of those can be considered as the basis of the bot). Also gating commits on passing a test run probably would also be good. > > I'm sure we will want to use some labels and milestones to track what PRs are for what versions, if they are blocked on something, etc. > > --- > > Long email! :) I think that is my current brain dumped in email form. As I said at the beginning, I think we should focus on what is blocking the easiest repos first and then just keep knocking down blockers as we try to move over more repos. > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Mon Jan 4 21:42:52 2016 From: barry at python.org (Barry Warsaw) Date: Mon, 4 Jan 2016 21:42:52 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition References: Message-ID: <20160104214252.4767eca7@anarchist.wooz.org> On Jan 05, 2016, at 12:42 AM, Brett Cannon wrote: >The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, >devguide, and cpython. Arthur: Each core dev converts four repos... Knight: Five repos Arthur: He who converts the repos four... Knight: Five repos Arthur: Five repos may hack in safety >There are six things to work out before we move over cpython. First, do we >want to split out Python 2 branches into their own repo? There might be a >clone size benefit which obviously is nice for people on slow Internet >connections. It also clearly separates out Python 2 from 3 and lets those >who prefer to focus on one compared to the other do that more easily. It >does potentially make any single fix that spans 2 and 3 require a bit more >work since it won't be an intra-clone change. We could also contemplate >sub-repos for things like the Doc/ or Tools/ directories (although I doubt >it's worth it). I suspect cherry-picking between Python 2 and 3 branches won't be very smooth no matter what approach is chosen. Still, it's probably just as easy to keep everything in one repo. >Second, do we want all fixes to go into master and then cherry-pick into >other branches IME, yes, that usually works the best for the person doing the cherry picking. It also seems that all contributed PRs are always just made against master anyway. >We should build a bot. It must use a Monty Python reference to trigger >(black-knight, none-shall-pass, man-from-scene-24, five-questions, >what-is-your-quest, what-is-your-favourite-colour, etc.; obviously I'm >leaning towards the Black Knight or Bridge of Death You've clearly done your homework! Which probably involved watching MPatHG at least 10 times. :) Something else to consider. We've long talked about splitting out the stdlib to make it easier for the alternative implementations to import. If some or all of them also switch to git, we could do that pretty easily with git submodules. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Mon Jan 4 21:46:27 2016 From: donald at stufft.io (Donald Stufft) Date: Mon, 4 Jan 2016 21:46:27 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: <20160104214252.4767eca7@anarchist.wooz.org> References: <20160104214252.4767eca7@anarchist.wooz.org> Message-ID: <810484FD-6243-4AD3-91FA-425DCF497FAC@stufft.io> > On Jan 4, 2016, at 9:42 PM, Barry Warsaw wrote: > > Something else to consider. We've long talked about splitting out the stdlib > to make it easier for the alternative implementations to import. If some or > all of them also switch to git, we could do that pretty easily with git > submodules. They could also use a subtree merge - https://jrsmith3.github.io/merging-a-subdirectory-from-another-repo-via-git-subtree.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From nicholas.chammas at gmail.com Mon Jan 4 21:50:12 2016 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Tue, 05 Jan 2016 02:50:12 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: <20160104214252.4767eca7@anarchist.wooz.org> References: <20160104214252.4767eca7@anarchist.wooz.org> Message-ID: Something else to consider. We?ve long talked about splitting out the stdlib to make it easier for the alternative implementations to import. If some or all of them also switch to git, we could do that pretty easily with git submodules. Not to derail here, but wasn?t there a discussion (perhaps on python-ideas) about slowly moving to a model where we distribute a barebones Python ?core?, allowing the standard modules to be updated and released on a more frequent cycle? Would this be one small step towards such a model? Nick ? On Mon, Jan 4, 2016 at 9:43 PM Barry Warsaw wrote: > On Jan 05, 2016, at 12:42 AM, Brett Cannon wrote: > > >The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, > >devguide, and cpython. > > Arthur: Each core dev converts four repos... > Knight: Five repos > Arthur: He who converts the repos four... > Knight: Five repos > Arthur: Five repos may hack in safety > > >There are six things to work out before we move over cpython. First, do we > >want to split out Python 2 branches into their own repo? There might be a > >clone size benefit which obviously is nice for people on slow Internet > >connections. It also clearly separates out Python 2 from 3 and lets those > >who prefer to focus on one compared to the other do that more easily. It > >does potentially make any single fix that spans 2 and 3 require a bit more > >work since it won't be an intra-clone change. We could also contemplate > >sub-repos for things like the Doc/ or Tools/ directories (although I doubt > >it's worth it). > > I suspect cherry-picking between Python 2 and 3 branches won't be very > smooth > no matter what approach is chosen. Still, it's probably just as easy to > keep > everything in one repo. > > >Second, do we want all fixes to go into master and then cherry-pick into > >other branches > > IME, yes, that usually works the best for the person doing the cherry > picking. > It also seems that all contributed PRs are always just made against master > anyway. > > >We should build a bot. It must use a Monty Python reference to trigger > >(black-knight, none-shall-pass, man-from-scene-24, five-questions, > >what-is-your-quest, what-is-your-favourite-colour, etc.; obviously I'm > >leaning towards the Black Knight or Bridge of Death > > You've clearly done your homework! Which probably involved watching > MPatHG at > least 10 times. :) > > Something else to consider. We've long talked about splitting out the > stdlib > to make it easier for the alternative implementations to import. If some > or > all of them also switch to git, we could do that pretty easily with git > submodules. > > Cheers, > -Barry > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jan 4 22:18:32 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Jan 2016 13:18:32 +1000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On 5 January 2016 at 10:42, Brett Cannon wrote: > So consider this the starting discussion of the PEP that will be the > hg.python.org -> GitHub transition PEP that I will be in charge of. Once we > have general agreement on the steps necessary I will start the actual PEP > and check it in, but I figure there's no point in have a skeleton PEP if we > can't agree on the skeleton. :) While I list steps influencing all the > repos, I want to focus on the ones stopping any repo from moving over for > now, expanding what we worry about to the cpython repo as we knock blockers > down until we move everything over and start adding GitHub perks. > > The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, > devguide, and cpython. I also think that's essentially the order we should > migrate them over. Some things will need to be universally handled before we > transition a single repo, while other things are only a blocker for some of > the repos. > > Universal blockers > ============== > There are four blockers that must be resolved before we even consider moving > a repo over. They can be solved in parallel, but they all need to have a > selected solution before we can move even the devinabox repo. > > First, we need to decide how we are going to handle adding all the core devs > to GitHub. Are we simply going to add all of them to the python > organization, or do we want something like a specific python-dev gteamroup > that gets added to all of the relevant repos? Basically I'm not sure how > picky we want to be about the people who already have organization access on > GitHub about them implicitly getting access to the cpython repo at the end > of the day (and the same goes for any of the other repos in the python > organization). For tracking names, I figure we will create a file in the > devguide where people can check in their GitHub usernames and I can manually > add people as people add themselves to the file. I think we want at least one group to track CPython commit privileges and potentially a second to track CLA signatories. For collecting GitHub username info, I think it makes more sense to add a new User profile field in Roundup than it does to use a text file in the devguide: http://roundup.sourceforge.net/docs/customizing.html#tracker-schema That way whether or not someone has signed the CLA and whether or not they have commit privileges is directly associated with both their Roundup user ID and their GitHub one, but the latter is only linked if they choose to provide it. It also means that if we eventually have a Roundup hook submitting patches on behalf of people (perhaps triggered by a "Create PR" link on the patch display rather than implicitly), it can set "Author" on the PR correctly if the author has provided a GitHub username. We should also consider the question of who needs to be added to the admin group for the GitHub python organisation. > Second, CLA enforcement. As of right now people go to > https://www.python.org/psf/contrib/contrib-form/, fill in the form, and then > Ewa gets an email where she manually flips a flag in Roundup. If we want to > use a web hook to verify someone has signed a CLA then we need to decide > where the ground truth for CLAs are. Do we want to keep using Roundup to > manage CLA agreements and thus add a GitHub field in bugs.python.org for > people's profile and a web hook or bot that will signal if someone has the > flag flipped on bugs.python.org? Or is there some prepackaged service that > we can use that will keep track of this which would cause us to not use > Roundup (which might be easier, but depending on the service require > everyone to re-sign)? There's also the issue of supporting people who want > to submit code by uploading a patch to bugs.python.org but not use GitHub. > Either way I don't want to have to ask everyone who submits a PR what their > bugs.python.org username is and then go check that manually. The way kubernetes does this is that googlebot checks if the submitter has signed the CLA, and if they have it sets a green "cla: yes" flag on the PR: https://github.com/kubernetes/kubernetes/labels/cla%3A%20yes If they haven't, then it posts a message asking them to sign it and applies a red "cla: no" label: https://github.com/kubernetes/kubernetes/pull/19271#issuecomment-168836357 For us, I think the approach that makes the most sense depends on whether or not it's easy to query the Roundup user DB based on a custom field. If it's easy, then I think we should just have the bot query Roundup to ask "Is this GitHub user id linked to a user account that has signed the CLA?". If querying by custom field is a pain, then I think we should instead have a GitHub group to track CLA signatories and tweak Roundup to add someone to that group with their signatory status is updated. The bot would then check the derived GitHub group rather than querying Roundup directly. > Fourth, for the ancillary repos of devinabox, peps, benchmarks, and > devguide, do we care if we use the GitHub merge button for PRs or do we want > to enforce a linear history with all repos? We just need to decide if care > about linear histories and then we can move forward since any bot we create > won't block us from using GitHub. Linear history is most useful for bisecting regressions, so I don't see a major need for it on any of the repos other than the main CPython one, while for at least the PEPs and the devguide I see a lot of value in enabling the low friction edit-PR-merge workflow for submitting small doc fixes (typos, etc). > Website-related stuff > ================ > This also almost gets us the peps repo, but we do need to figure out how to > change the website to build from the git checkout rather than an hg one. > Same goes for the devguide. It would be great if we can set up web hooks to > immediately trigger rebuilds of those portions of the sites instead of > having to wait until a cronjob triggers. I like Donald's suggestion of setting up a webhook to ensure hg.python.org remains an up to date read-only Mercurial mirror. (Which suggests another important step: tweaking the configuration of those repos to block commits as part of the cutover of each repo) Something we may also want to consider is whether or not we might be able to use ReadTheDocs for building and hosting at least some of the ancillary repos (it would be nice to be able to use full Sphinx markup when writing PEPs, for example, and the devguide is already a Sphinx project). > CPython requirements > ================= > There are six things to work out before we move over cpython. First, do we > want to split out Python 2 branches into their own repo? There might be a > clone size benefit which obviously is nice for people on slow Internet > connections. Shallow clones are going to be more of a benefit there, and I agree with Donald that splitting the repos would be more trouble than it's worth. > We could also contemplate > sub-repos for things like the Doc/ or Tools/ directories (although I doubt > it's worth it). While I'd still like to see the tutorial and the HOWTO guides moved out to their own version independent repos in the long term, I don't see any reason to do that as part of the migration to a richer repository hosting environment - we can consider if and when we want to tackle it *after* the migration. > Second, do we want all fixes to go into master and then cherry-pick into > other branches, or do we want to follow our current practice of going into > the active feature branch and then merge into master? I personally prefer > the former and I think most everyone else does as well, but I thought it > should be at least thought about. Master+cherry-pick makes sense to me. I have two concrete reasons for that, one personal and one professional: * the personal reason is that it means I can effectively ignore all but the most recent maintenance branch when contributing on my own time * the professional reason is that "please backport the fix for issue #ABC to version X.Y" requests become much easier to handle (even if X.Y is in security fix only mode), as it aligns with the normal upstream workflow rather than being an exceptional case > Third, how to handle Misc/NEWS? We can add a NEWS field to bugs.python.org > and then generate the NEWS file by what issues are tied to what version and > when they were closed. The other approach is to create a file per NEWS entry > in a version-specific directory (Larry created code for hg already for this > to give people an idea: http://bugs.python.org/issue18967). Then when we cut > a release we run a tool the slurps up all of the relevant files -- which > includes files in the directory for the next feature release which represent > fixes which were cherry picked -- and generates the NEWS file for the final > release. The per-file approach is bot-friendly and also CLI-friendly, but > potentially requires more tooling and I don't know if people feel news > entries should be tied to the issue or in the repo (although that assumes > people find tweaking Roundup easy :). I'm generally a fan of loose coupling, and the main virtue I see for "in the repo" file-based approaches is that it means that everything you need to generate the release notes remains with the code and documentation - you're not relying on an external service being available. It also doesn't rule out the use of a tracker field later - such a field would just need to be converted into a file in the repo when preparing the patch. > Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we > flatten them, I believe the PR creators will get credit in the commit as the > author while the core dev committing will be flagged as the person doing the > merge (someone correct me if I'm wrong because if I am this whole point is > silly). With the commits containing credit directly, we can either > automatically generate Misc/ACKS like the NEWS file or simply drop it for > future contributors and just leave the file for past contributors since git > will have kept track for us. I think we'll still need the manual ACKS file to accommodate patches uploaded to Roundup (especially older ones), so I think the most useful thing to do here is to have a script that can check the ACKs file for missing names that appear in the GitHub contributor list. > We should build a bot. It must use a Monty Python reference to trigger > (black-knight, none-shall-pass, man-from-scene-24, five-questions, > what-is-your-quest, what-is-your-favourite-colour, etc.; obviously I'm > leaning towards the Black Knight or Bridge of Death scenes from the Holy > Grail for inspiration since they deal with blocking you from doing > something). I'm still a fan of "blue-no-green" :) > It should handle specifying the commit message, what branches to > commit to/cherry pick into, and a NEWS entry (if necessary). I don't know if > it needs to do anything else as a requirement. It should probably implement > a commit queue like Zuul or Homu (and both of those can be considered as the > basis of the bot). Also gating commits on passing a test run probably would > also be good. Something else worth considering is whether to have one bot or multiple. With the Kubernetes issue I linked above for example, you can see that the googlebot handles the CLA question, but there's a separate k8s-bot that prompts committers to move the issue along (and what appears to be a third party bot doing integration testing for Mesosphere). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jan 4 22:38:43 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Jan 2016 13:38:43 +1000 Subject: [core-workflow] Standard library separation from core (was Re: My initial thoughts on the steps/blockers of the transition) Message-ID: On 5 January 2016 at 12:50, Nicholas Chammas wrote: > Something else to consider. We?ve long talked about splitting out the stdlib > to make it easier for the alternative implementations to import. If some or > all of them also switch to git, we could do that pretty easily with git > submodules. > > Not to derail here, but wasn?t there a discussion (perhaps on python-ideas) > about slowly moving to a model where we distribute a barebones Python > ?core?, allowing the standard modules to be updated and released on a more > frequent cycle? Would this be one small step towards such a model? That discussion has been going on for years :) The most extensive elaboration is in the related PEPs: PEP 407 considered the idea of distinguishing normal releases and LTS releases: https://www.python.org/dev/peps/pep-0407/ PEP 413 considered decoupling standard library versions from language versions: https://www.python.org/dev/peps/pep-0413/ The ripple effect of either proposal on the wider community would have been huge though, hence why 407 is Deferred and 413 Withdrawn. Instead, the main step which has been taken (driven in no small part by the Python 3 transition) is the creation of PyPI counterparts for modules that see substantial updates that are backwards compatible with earlier versions (importlib2, for example, lets you use the Python 3 import system in Python 2). Shipping pip by default with the interpreter runtime is also pushing people more towards the notion that "if you're limiting yourself to the standard library, you're experiencing only a fraction of what the Python ecosystem has to offer you". We don't currently do a great job of making those libraries *discoverable* by end users, but they're available if you know to look for them (there's an incomplete list at https://wiki.python.org/moin/Python2orPython3#Supporting_Python_2_and_Python_3_in_a_common_code_base ) pip's inclusion was also the first instance of CPython shipping a *bundled* library that isn't maintained through the CPython development process - each new maintenance release of CPython ships the latest upstream version of pip, rather than being locked to the version of pip that shipped with the corresponding x.y.0 release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jan 4 22:45:30 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Jan 2016 13:45:30 +1000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: <93574A50-8E31-49D1-80A4-590AC4C347D7@stufft.io> References: <93574A50-8E31-49D1-80A4-590AC4C347D7@stufft.io> Message-ID: On 5 January 2016 at 11:08, Donald Stufft wrote: > > On Jan 4, 2016, at 7:42 PM, Brett Cannon wrote: >> We should try to get test coverage wired up as well per CI. I don't know if >> coveralls.io or some other provider is best, but we should see what is >> available and find out if we can use them to either get basic coverage or >> thorough coverage (read https://hg.python.org/devinabox/file/tip/README#l124 >> to see what thorough coverage entails, but it does require a checkout of >> coverage.py). > > I prefer codecov, but it shouldn?t be too hard to do. I tried to get Python > + C coverage checking in the demo with that, but I failed at making the C > coverage work. Another posslble tool worth considering is diff_cover: https://pypi.python.org/pypi/diff_cover/ That uses git diff to find the lines affected by a patch and specifically looks up *those lines* in a coverage report, so it can ensure that any lines changed by a patch are covered by the regression test suite. It appears to be a neat way of guiding a code base towards more comprehensive test coverage. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From nicholas.chammas at gmail.com Mon Jan 4 23:14:43 2016 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Tue, 05 Jan 2016 04:14:43 +0000 Subject: [core-workflow] Standard library separation from core (was Re: My initial thoughts on the steps/blockers of the transition) In-Reply-To: References: Message-ID: Thanks for sharing that background, Nick. Instead, the main step which has been taken (driven in no small part by the Python 3 transition) is the creation of PyPI counterparts for modules that see substantial updates that are backwards compatible with earlier versions (importlib2, for example, lets you use the Python 3 import system in Python 2). So is the intention that, over the long term, these PyPI counterparts would cannibalize their standard library equivalents in terms of usage? Nick ? On Mon, Jan 4, 2016 at 10:38 PM Nick Coghlan wrote: > On 5 January 2016 at 12:50, Nicholas Chammas > wrote: > > Something else to consider. We?ve long talked about splitting out the > stdlib > > to make it easier for the alternative implementations to import. If some > or > > all of them also switch to git, we could do that pretty easily with git > > submodules. > > > > Not to derail here, but wasn?t there a discussion (perhaps on > python-ideas) > > about slowly moving to a model where we distribute a barebones Python > > ?core?, allowing the standard modules to be updated and released on a > more > > frequent cycle? Would this be one small step towards such a model? > > That discussion has been going on for years :) > > The most extensive elaboration is in the related PEPs: > > PEP 407 considered the idea of distinguishing normal releases and LTS > releases: https://www.python.org/dev/peps/pep-0407/ > PEP 413 considered decoupling standard library versions from language > versions: https://www.python.org/dev/peps/pep-0413/ > > The ripple effect of either proposal on the wider community would have > been huge though, hence why 407 is Deferred and 413 Withdrawn. > > Instead, the main step which has been taken (driven in no small part > by the Python 3 transition) is the creation of PyPI counterparts for > modules that see substantial updates that are backwards compatible > with earlier versions (importlib2, for example, lets you use the > Python 3 import system in Python 2). Shipping pip by default with the > interpreter runtime is also pushing people more towards the notion > that "if you're limiting yourself to the standard library, you're > experiencing only a fraction of what the Python ecosystem has to offer > you". > > We don't currently do a great job of making those libraries > *discoverable* by end users, but they're available if you know to look > for them (there's an incomplete list at > > https://wiki.python.org/moin/Python2orPython3#Supporting_Python_2_and_Python_3_in_a_common_code_base > ) > > pip's inclusion was also the first instance of CPython shipping a > *bundled* library that isn't maintained through the CPython > development process - each new maintenance release of CPython ships > the latest upstream version of pip, rather than being locked to the > version of pip that shipped with the corresponding x.y.0 release. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jan 4 23:18:43 2016 From: donald at stufft.io (Donald Stufft) Date: Mon, 4 Jan 2016 23:18:43 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <93574A50-8E31-49D1-80A4-590AC4C347D7@stufft.io> Message-ID: <30D0F8AB-AD34-43A3-A5CE-DB41C54F0C33@stufft.io> > On Jan 4, 2016, at 10:45 PM, Nick Coghlan wrote: > > On 5 January 2016 at 11:08, Donald Stufft wrote: >> >> On Jan 4, 2016, at 7:42 PM, Brett Cannon wrote: >>> We should try to get test coverage wired up as well per CI. I don't know if >>> coveralls.io or some other provider is best, but we should see what is >>> available and find out if we can use them to either get basic coverage or >>> thorough coverage (read https://hg.python.org/devinabox/file/tip/README#l124 >>> to see what thorough coverage entails, but it does require a checkout of >>> coverage.py). >> >> I prefer codecov, but it shouldn?t be too hard to do. I tried to get Python >> + C coverage checking in the demo with that, but I failed at making the C >> coverage work. > > Another posslble tool worth considering is diff_cover: > https://pypi.python.org/pypi/diff_cover/ > > That uses git diff to find the lines affected by a patch and > specifically looks up *those lines* in a coverage report, so it can > ensure that any lines changed by a patch are covered by the regression > test suite. It appears to be a neat way of guiding a code base towards > more comprehensive test coverage. > FWIW codecov has that built in. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Tue Jan 5 00:22:35 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Jan 2016 15:22:35 +1000 Subject: [core-workflow] Standard library separation from core (was Re: My initial thoughts on the steps/blockers of the transition) In-Reply-To: References: Message-ID: On 5 January 2016 at 14:14, Nicholas Chammas wrote: > Thanks for sharing that background, Nick. > > Instead, the main step which has been taken (driven in no small part > by the Python 3 transition) is the creation of PyPI counterparts for > modules that see substantial updates that are backwards compatible > with earlier versions (importlib2, for example, lets you use the > Python 3 import system in Python 2). > > So is the intention that, over the long term, these PyPI counterparts would > cannibalize their standard library equivalents in terms of usage? Probably not - the baseline versions will almost certainly always be used more heavily simply due to being available by default. What the PyPI releases mean is that the folks for whom the standard library version is old enough to be annoying now have the freedom to choose between selectively updating just that component and upgrading to a new version of the language runtime, and the former is important when you don't have full control over the target runtime environment (e.g. many folks are paid to support the system Python runtimes on various versions of Linux, and only drop support for those old versions when the Linux vendors do). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ezio.melotti at gmail.com Tue Jan 5 00:53:54 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Tue, 5 Jan 2016 07:53:54 +0200 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, Jan 5, 2016 at 2:42 AM, Brett Cannon wrote: > So consider this the starting discussion of the PEP that will be the > hg.python.org -> GitHub transition PEP that I will be in charge of. Once we > have general agreement on the steps necessary I will start the actual PEP > and check it in, but I figure there's no point in have a skeleton PEP if we > can't agree on the skeleton. :) While I list steps influencing all the > repos, I want to focus on the ones stopping any repo from moving over for > now, expanding what we worry about to the cpython repo as we knock blockers > down until we move everything over and start adding GitHub perks. > > The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, > devguide, and cpython. On top of this, there is also the test repo (https://hg.python.org/test) and all the tracker repos (https://hg.python.org/tracker/). I think it would be useful to port the former since it will provide a place for devs to try things out and experiment (a new test repo could also be created though). It would be nice to port the tracker repos too and be consistent with the others, but it's not a priority. When we switched to HG they kept being on SVN until I ported them, so I guess the same thing can be done (unless R. David or Martin prefer to stick to HG). > I also think that's essentially the order we should > migrate them over. Some things will need to be universally handled before we > transition a single repo, while other things are only a blocker for some of > the repos. > > Universal blockers > ============== > There are four blockers that must be resolved before we even consider moving > a repo over. They can be solved in parallel, but they all need to have a > selected solution before we can move even the devinabox repo. > > First, we need to decide how we are going to handle adding all the core devs > to GitHub. Are we simply going to add all of them to the python > organization, or do we want something like a specific python-dev gteamroup > that gets added to all of the relevant repos? Basically I'm not sure how > picky we want to be about the people who already have organization access on > GitHub about them implicitly getting access to the cpython repo at the end > of the day (and the same goes for any of the other repos in the python > organization). For tracking names, I figure we will create a file in the > devguide where people can check in their GitHub usernames and I can manually > add people as people add themselves to the file. > I think the current list of core-devs should be converted to a group and given access to the same repos they have access to now (i.e. cpython/devguide/peps and possibly others). Then additional repo-specific groups can be created in case we want to let specific contributors work on peps or the devguide. > Second, CLA enforcement. As of right now people go to > https://www.python.org/psf/contrib/contrib-form/, fill in the form, and then > Ewa gets an email where she manually flips a flag in Roundup. If we want to > use a web hook to verify someone has signed a CLA then we need to decide > where the ground truth for CLAs are. Do we want to keep using Roundup to > manage CLA agreements and thus add a GitHub field in bugs.python.org for > people's profile and a web hook or bot that will signal if someone has the > flag flipped on bugs.python.org? This can be done. We can add a "GitHub" username field to Roundup users so that we can link the two. > Or is there some prepackaged service that > we can use that will keep track of this which would cause us to not use > Roundup (which might be easier, but depending on the service require > everyone to re-sign)? There's also the issue of supporting people who want > to submit code by uploading a patch to bugs.python.org but not use GitHub. > Either way I don't want to have to ask everyone who submits a PR what their > bugs.python.org username is and then go check that manually. > This also brings up another problem. Since the discussion about an issue happens on b.p.o and the PRs are submitted on GitHub, this means that: 1) users with only a GitHub account have to create a b.p.o account if they want to comment on the issue (exclusing review comments); 2) users with only a b.p.o account have to create a GitHub account if they want to review a PR; 3) users with both can comment on b.p.o and review on GitHub, but they might need to login twice. It would be better if users didn't need to create and use two separate accounts. > Third, how do we want to do the repo conversions? We need to choose the > tool(s) and command(s) that we want to use. There was mention of wanting a > mapping from hg commit ID to git commit ID. If we have that we could have a > static bugs.python.org/commit/ page that had the mapping embedded in > some JavaScript and if matched then we just forward them to the > corresponding GitHub commit page, otherwise just blindly forward to GitHub > and assume the ID is git-only, giving us a stable URL for commit web views. > As I mentioned on python-committers, we already have https://hg.python.org/lookup/ . This is currently used to map SVN->HG (e.g. https://hg.python.org/lookup/r12345 ), and should be extended to handle cs ids too. The b.p.o linkifier can just convert all revision numbers and cs ids to a https://hg.python.org/lookup/ link and let the lookup page figure out where to redirect the user. > Fourth, for the ancillary repos of devinabox, peps, benchmarks, and > devguide, do we care if we use the GitHub merge button for PRs or do we want > to enforce a linear history with all repos? We just need to decide if care > about linear histories and then we can move forward since any bot we create > won't block us from using GitHub. > > Those four things are enough to move devinabox over. It probably is enough > for the benchmarks suite, but I have an email to speed@ asking if people > want to use this opportunity to re-evaluate the benchmark suite and make any > changes that will affect repo size (e.g., use pip to pull in the libraries > and frameworks used by a benchmark rather than vendoring their code, making > the repo much smaller). > > Website-related stuff > ================ > This also almost gets us the peps repo, but we do need to figure out how to > change the website to build from the git checkout rather than an hg one. > Same goes for the devguide. It would be great if we can set up web hooks to > immediately trigger rebuilds of those portions of the sites instead of > having to wait until a cronjob triggers. > I think we should make hg.python.org read-only but keep it around and in sync with the GitHub repo (either via cronjobs or hooks). This will allow people to contribute using HG in the same way that the current GitHub clone allows people to contribute using git. It will also avoid breaking all the tools that currently use hg.python.org (and buys us more time to port them if/when needed). > CPython requirements > ================= > There are six things to work out before we move over cpython. First, do we > want to split out Python 2 branches into their own repo? There might be a > clone size benefit which obviously is nice for people on slow Internet > connections. It also clearly separates out Python 2 from 3 and lets those > who prefer to focus on one compared to the other do that more easily. It > does potentially make any single fix that spans 2 and 3 require a bit more > work since it won't be an intra-clone change. We could also contemplate > sub-repos for things like the Doc/ or Tools/ directories (although I doubt > it's worth it). > I think we should keep 2/3 together. We could split the stdlib from the rest, but that's a separate issue. > Second, do we want all fixes to go into master and then cherry-pick into > other branches, or do we want to follow our current practice of going into > the active feature branch and then merge into master? I personally prefer > the former and I think most everyone else does as well, but I thought it > should be at least thought about. > Master first and cherry-picking for older branches sounds good to me, but I don't know if switching model will have any implications, especially while going through the history or using tools like bisect. > Third, how to handle Misc/NEWS? We can add a NEWS field to bugs.python.org > and then generate the NEWS file by what issues are tied to what version and > when they were closed. The other approach is to create a file per NEWS entry > in a version-specific directory (Larry created code for hg already for this > to give people an idea: http://bugs.python.org/issue18967). Then when we cut > a release we run a tool the slurps up all of the relevant files -- which > includes files in the directory for the next feature release which represent > fixes which were cherry picked -- and generates the NEWS file for the final > release. The per-file approach is bot-friendly and also CLI-friendly, but > potentially requires more tooling and I don't know if people feel news > entries should be tied to the issue or in the repo (although that assumes > people find tweaking Roundup easy :). > > Fourth, we need to decide exactly what commands we expect core devs to run > initially for committing code. Since we agreed to a linear history we need > to document exactly what we expect people to do for a PR to get it into > their git repo. This will go into the devguide -- probably will want to > start a github branch at some point -- and will act as the commands the bot > will want to work off of. > I would like to see a complete list of steps from starting to work on an issue to having it in the repo, at least to understand the new workflow. This doesn't have to include all the specific commands, but at least the basic steps (e.g. after I made a patch to I commit it and send a pull request to the main repo, or do I push it to my GitHub clone and push a button to send the PR? Do I need to create a branch before I start working on an issue? > Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we > flatten them, I believe the PR creators will get credit in the commit as the > author while the core dev committing will be flagged as the person doing the > merge (someone correct me if I'm wrong because if I am this whole point is > silly). With the commits containing credit directly, we can either > automatically generate Misc/ACKS like the NEWS file or simply drop it for > future contributors and just leave the file for past contributors since git > will have kept track for us. > We could keep updating for regular patches with no related PR and add a note about all the other GIT contributors (possibly with a git command that lists all authors). Later on we might decide to have a script that automatically adds all the GIT contributors automatically. > Six, we will need to update our Buildbot fleet. > If we keep hg.p.o around and updated, we might not have to do this now (even though now is better than never). > This gets us to the bare minimum needed to function. > > Parity with hg.python.org > ---------------------------------- > For parity, there are some Roundup integrations that will be necessary, like > auto-generating links, posting commits to #python-dev on IRC, etc. I'm not > sure if people want to block until that is all in place or not. I do think > we should make sure there is some web hook that can take an issue # from the > title of a PR and automatically posts to the corresponding issue on > bugs.python.org that the PR exists. If people disagree then feel free to say > so. > FWIW I started adding notes to https://wiki.python.org/moin/TrackerDevelopmentPlanning to track everything that needs to be done on the Roundup side. If you prefer I can later move this to the new PEP, but for now I'm using it to keep track of all the things that come up in the various threads. Best Regards, Ezio Melotti > Adding perks > ========== > ... From ezio.melotti at gmail.com Tue Jan 5 00:58:08 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Tue, 5 Jan 2016 07:58:08 +0200 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, Jan 5, 2016 at 5:18 AM, Nick Coghlan wrote: > On 5 January 2016 at 10:42, Brett Cannon wrote: >> ... >> >> First, we need to decide how we are going to handle adding all the core devs >> to GitHub. Are we simply going to add all of them to the python >> organization, or do we want something like a specific python-dev gteamroup >> that gets added to all of the relevant repos? Basically I'm not sure how >> picky we want to be about the people who already have organization access on >> GitHub about them implicitly getting access to the cpython repo at the end >> of the day (and the same goes for any of the other repos in the python >> organization). For tracking names, I figure we will create a file in the >> devguide where people can check in their GitHub usernames and I can manually >> add people as people add themselves to the file. > > I think we want at least one group to track CPython commit privileges > and potentially a second to track CLA signatories. > > For collecting GitHub username info, I think it makes more sense to > add a new User profile field in Roundup than it does to use a text > file in the devguide: > http://roundup.sourceforge.net/docs/customizing.html#tracker-schema > Agreed. > That way whether or not someone has signed the CLA and whether or not > they have commit privileges is directly associated with both their > Roundup user ID and their GitHub one, but the latter is only linked if > they choose to provide it. > > It also means that if we eventually have a Roundup hook submitting > patches on behalf of people (perhaps triggered by a "Create PR" link > on the patch display rather than implicitly), it can set "Author" on > the PR correctly if the author has provided a GitHub username. > > We should also consider the question of who needs to be added to the > admin group for the GitHub python organisation. > >> Second, CLA enforcement. As of right now people go to >> https://www.python.org/psf/contrib/contrib-form/, fill in the form, and then >> Ewa gets an email where she manually flips a flag in Roundup. If we want to >> use a web hook to verify someone has signed a CLA then we need to decide >> where the ground truth for CLAs are. Do we want to keep using Roundup to >> manage CLA agreements and thus add a GitHub field in bugs.python.org for >> people's profile and a web hook or bot that will signal if someone has the >> flag flipped on bugs.python.org? Or is there some prepackaged service that >> we can use that will keep track of this which would cause us to not use >> Roundup (which might be easier, but depending on the service require >> everyone to re-sign)? There's also the issue of supporting people who want >> to submit code by uploading a patch to bugs.python.org but not use GitHub. >> Either way I don't want to have to ask everyone who submits a PR what their >> bugs.python.org username is and then go check that manually. > > The way kubernetes does this is that googlebot checks if the submitter > has signed the CLA, and if they have it sets a green "cla: yes" flag > on the PR: https://github.com/kubernetes/kubernetes/labels/cla%3A%20yes > > If they haven't, then it posts a message asking them to sign it and > applies a red "cla: no" label: > https://github.com/kubernetes/kubernetes/pull/19271#issuecomment-168836357 > > For us, I think the approach that makes the most sense depends on > whether or not it's easy to query the Roundup user DB based on a > custom field. If it's easy, then I think we should just have the bot > query Roundup to ask "Is this GitHub user id linked to a user account > that has signed the CLA?". > It should be easy. Best Regards, Ezio Melotti > If querying by custom field is a pain, then I think we should instead > have a GitHub group to track CLA signatories and tweak Roundup to add > someone to that group with their signatory status is updated. The bot > would then check the derived GitHub group rather than querying Roundup > directly. > From donald at stufft.io Tue Jan 5 06:08:18 2016 From: donald at stufft.io (Donald Stufft) Date: Tue, 5 Jan 2016 06:08:18 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: <32CAF60C-9837-404F-9862-48F95DB15764@stufft.io> > On Jan 5, 2016, at 12:53 AM, Ezio Melotti wrote: > >> >> Or is there some prepackaged service that >> we can use that will keep track of this which would cause us to not use >> Roundup (which might be easier, but depending on the service require >> everyone to re-sign)? There's also the issue of supporting people who want >> to submit code by uploading a patch to bugs.python.org but not use GitHub. >> Either way I don't want to have to ask everyone who submits a PR what their >> bugs.python.org username is and then go check that manually. >> > > This also brings up another problem. > Since the discussion about an issue happens on b.p.o and the PRs are > submitted on GitHub, this means that: > 1) users with only a GitHub account have to create a b.p.o account if > they want to comment on the issue (exclusing review comments); > 2) users with only a b.p.o account have to create a GitHub account if > they want to review a PR; > 3) users with both can comment on b.p.o and review on GitHub, but they > might need to login twice. > > It would be better if users didn't need to create and use two separate accounts. Can we allow (not mandate) login to b.p.o with a GitHub account? That should solve #1 and should sort of solve #3 as well. Ideally it?d allow you to associate a Github account with an existing b.p.o account (and doing so should also automatically create an association that X on b.p.o is Y on Github). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mal at egenix.com Tue Jan 5 06:22:03 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 5 Jan 2016 12:22:03 +0100 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: <568BA75B.6000106@egenix.com> On 05.01.2016 06:53, Ezio Melotti wrote: >> Or is there some prepackaged service that >> we can use that will keep track of this which would cause us to not use >> Roundup (which might be easier, but depending on the service require >> everyone to re-sign)? There's also the issue of supporting people who want >> to submit code by uploading a patch to bugs.python.org but not use GitHub. >> Either way I don't want to have to ask everyone who submits a PR what their >> bugs.python.org username is and then go check that manually. >> > > This also brings up another problem. > Since the discussion about an issue happens on b.p.o and the PRs are > submitted on GitHub, this means that: > 1) users with only a GitHub account have to create a b.p.o account if > they want to comment on the issue (exclusing review comments); > 2) users with only a b.p.o account have to create a GitHub account if > they want to review a PR; > 3) users with both can comment on b.p.o and review on GitHub, but they > might need to login twice. > > It would be better if users didn't need to create and use two separate accounts. Given that we want to make it possible to move away from Github without too much fuzz, wouldn't it be better to have the PR discussions on b.p.o and Rietvield ? If we start using Github for this, we'd lose that part of the review history when moving away. Moving from the SF issue tracker to b.p.o was a major piece of work (mostly done by Martin von L?wis IIRC) and it's not clear how we could retrofit those discussions into the b.p.o issue discussions. Perhaps we could gateway the emails that Github sends for PR discussions back into b.p.o in some way (the emails contain the PR number, repo name and Github account name of the sender in the headers). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jan 05 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From brett at python.org Tue Jan 5 12:13:09 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 17:13:09 +0000 Subject: [core-workflow] Standard library separation from core (was Re: My initial thoughts on the steps/blockers of the transition) In-Reply-To: References: Message-ID: On Mon, 4 Jan 2016 at 21:22 Nick Coghlan wrote: > On 5 January 2016 at 14:14, Nicholas Chammas > wrote: > > Thanks for sharing that background, Nick. > > > > Instead, the main step which has been taken (driven in no small part > > by the Python 3 transition) is the creation of PyPI counterparts for > > modules that see substantial updates that are backwards compatible > > with earlier versions (importlib2, for example, lets you use the > > Python 3 import system in Python 2). > > > > So is the intention that, over the long term, these PyPI counterparts > would > > cannibalize their standard library equivalents in terms of usage? > > Probably not - the baseline versions will almost certainly always be > used more heavily simply due to being available by default. > > What the PyPI releases mean is that the folks for whom the standard > library version is old enough to be annoying now have the freedom to > choose between selectively updating just that component and upgrading > to a new version of the language runtime, and the former is important > when you don't have full control over the target runtime environment > (e.g. many folks are paid to support the system Python runtimes on > various versions of Linux, and only drop support for those old > versions when the Linux vendors do). If you guys wants to continue this conversation, the stdlib-sig is the perfect place to have this discussion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 5 12:50:53 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 17:50:53 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: <568BA75B.6000106@egenix.com> References: <568BA75B.6000106@egenix.com> Message-ID: On Tue, 5 Jan 2016 at 03:22 M.-A. Lemburg wrote: > On 05.01.2016 06:53, Ezio Melotti wrote: > >> Or is there some prepackaged service that > >> we can use that will keep track of this which would cause us to not use > >> Roundup (which might be easier, but depending on the service require > >> everyone to re-sign)? There's also the issue of supporting people who > want > >> to submit code by uploading a patch to bugs.python.org but not use > GitHub. > >> Either way I don't want to have to ask everyone who submits a PR what > their > >> bugs.python.org username is and then go check that manually. > >> > > > > This also brings up another problem. > > Since the discussion about an issue happens on b.p.o and the PRs are > > submitted on GitHub, this means that: > > 1) users with only a GitHub account have to create a b.p.o account if > > they want to comment on the issue (exclusing review comments); > > 2) users with only a b.p.o account have to create a GitHub account if > > they want to review a PR; > > 3) users with both can comment on b.p.o and review on GitHub, but they > > might need to login twice. > > > > It would be better if users didn't need to create and use two separate > accounts. > > Given that we want to make it possible to move away from Github > without too much fuzz, wouldn't it be better to have the > PR discussions on b.p.o and Rietvield ? > One of the motivating factors of this move is to get off of our fork of Rietveld, so that's not feasible. > > If we start using Github for this, we'd lose that part of the > review history when moving away. > GitHub's API allows for exporting all of the data. > > Moving from the SF issue tracker to b.p.o was a major piece of work > (mostly done by Martin von L?wis IIRC) and it's not clear how we > could retrofit those discussions into the b.p.o issue discussions. > > Perhaps we could gateway the emails that Github sends for PR > discussions back into b.p.o in some way (the emails contain the > PR number, repo name and Github account name of the sender > in the headers). > I believe GitLab has a GitHub -> GitLab migration tool that keeps PR history, so this isn't quite the issue that it initially appears to be. If people are that worried, we could do a daily dump of the data. But unless we can have all GitHub-related comments to an issue not trigger an email update I don't think that's feasible as it would mean two emails for every PR comment; one from GitHub and one from b.p.o. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 5 13:03:18 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 18:03:18 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Mon, 4 Jan 2016 at 21:54 Ezio Melotti wrote: > On Tue, Jan 5, 2016 at 2:42 AM, Brett Cannon wrote: > > So consider this the starting discussion of the PEP that will be the > > hg.python.org -> GitHub transition PEP that I will be in charge of. > Once we > > have general agreement on the steps necessary I will start the actual PEP > > and check it in, but I figure there's no point in have a skeleton PEP if > we > > can't agree on the skeleton. :) While I list steps influencing all the > > repos, I want to focus on the ones stopping any repo from moving over for > > now, expanding what we worry about to the cpython repo as we knock > blockers > > down until we move everything over and start adding GitHub perks. > > > > The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, > > devguide, and cpython. > > On top of this, there is also the test repo > (https://hg.python.org/test) and all the tracker repos > (https://hg.python.org/tracker/). > I think it would be useful to port the former since it will provide a > place for devs to try things out and experiment (a new test repo could > also be created though). > It would be nice to port the tracker repos too and be consistent with > the others, but it's not a priority. When we switched to HG they kept > being on SVN until I ported them, so I guess the same thing can be > done (unless R. David or Martin prefer to stick to HG). > > > I also think that's essentially the order we should > > migrate them over. Some things will need to be universally handled > before we > > transition a single repo, while other things are only a blocker for some > of > > the repos. > > > > Universal blockers > > ============== > > There are four blockers that must be resolved before we even consider > moving > > a repo over. They can be solved in parallel, but they all need to have a > > selected solution before we can move even the devinabox repo. > > > > First, we need to decide how we are going to handle adding all the core > devs > > to GitHub. Are we simply going to add all of them to the python > > organization, or do we want something like a specific python-dev > gteamroup > > that gets added to all of the relevant repos? Basically I'm not sure how > > picky we want to be about the people who already have organization > access on > > GitHub about them implicitly getting access to the cpython repo at the > end > > of the day (and the same goes for any of the other repos in the python > > organization). For tracking names, I figure we will create a file in the > > devguide where people can check in their GitHub usernames and I can > manually > > add people as people add themselves to the file. > > > > I think the current list of core-devs should be converted to a group > and given access to the same repos they have access to now (i.e. > cpython/devguide/peps and possibly others). Then additional > repo-specific groups can be created in case we want to let specific > contributors work on peps or the devguide. > This seems to be the general consensus, so we will create a python-dev team under the python org and add the core devs there. > > > Second, CLA enforcement. As of right now people go to > > https://www.python.org/psf/contrib/contrib-form/, fill in the form, and > then > > Ewa gets an email where she manually flips a flag in Roundup. If we want > to > > use a web hook to verify someone has signed a CLA then we need to decide > > where the ground truth for CLAs are. Do we want to keep using Roundup to > > manage CLA agreements and thus add a GitHub field in bugs.python.org for > > people's profile and a web hook or bot that will signal if someone has > the > > flag flipped on bugs.python.org? > > This can be done. We can add a "GitHub" username field to Roundup > users so that we can link the two. > OK, so it sounds like we will stick with our current CLA signing flow and write our own CLA bot that will query Roundup as to whether someone has signed the CLA or not and then throw up a banner signalling if someone has (not) signed and an appropriate link to the CLA. That will require some Roundup work and the creation of the bot. I should also mention, any bot creations we do should abstract out the code review tool so that when we change providers again in the future it will be more straight-forward to just update some select APIs rather than rewrite every bot we create. > > > > Or is there some prepackaged service that > > we can use that will keep track of this which would cause us to not use > > Roundup (which might be easier, but depending on the service require > > everyone to re-sign)? There's also the issue of supporting people who > want > > to submit code by uploading a patch to bugs.python.org but not use > GitHub. > > Either way I don't want to have to ask everyone who submits a PR what > their > > bugs.python.org username is and then go check that manually. > > > > This also brings up another problem. > Since the discussion about an issue happens on b.p.o and the PRs are > submitted on GitHub, this means that: > 1) users with only a GitHub account have to create a b.p.o account if > they want to comment on the issue (exclusing review comments); > 2) users with only a b.p.o account have to create a GitHub account if > they want to review a PR; > 3) users with both can comment on b.p.o and review on GitHub, but they > might need to login twice. > > It would be better if users didn't need to create and use two separate > accounts. > If we can add GitHub as a login/creation option for b.p.o accounts then that solves that. But I'm willing to bet a majority of people will already have a GitHub account and we have always required the b.p.o account so #1 is the going to be the common case. > > > Third, how do we want to do the repo conversions? We need to choose the > > tool(s) and command(s) that we want to use. There was mention of wanting > a > > mapping from hg commit ID to git commit ID. If we have that we could > have a > > static bugs.python.org/commit/ page that had the mapping embedded in > > some JavaScript and if matched then we just forward them to the > > corresponding GitHub commit page, otherwise just blindly forward to > GitHub > > and assume the ID is git-only, giving us a stable URL for commit web > views. > > > > As I mentioned on python-committers, we already have > https://hg.python.org/lookup/ . > This is currently used to map SVN->HG (e.g. > https://hg.python.org/lookup/r12345 ), and should be extended to > handle cs ids too. > The b.p.o linkifier can just convert all revision numbers and cs ids > to a https://hg.python.org/lookup/ link and let the lookup page figure > out where to redirect the user. > > > Fourth, for the ancillary repos of devinabox, peps, benchmarks, and > > devguide, do we care if we use the GitHub merge button for PRs or do we > want > > to enforce a linear history with all repos? We just need to decide if > care > > about linear histories and then we can move forward since any bot we > create > > won't block us from using GitHub. > > > > Those four things are enough to move devinabox over. It probably is > enough > > for the benchmarks suite, but I have an email to speed@ asking if people > > want to use this opportunity to re-evaluate the benchmark suite and make > any > > changes that will affect repo size (e.g., use pip to pull in the > libraries > > and frameworks used by a benchmark rather than vendoring their code, > making > > the repo much smaller). > > > > Website-related stuff > > ================ > > This also almost gets us the peps repo, but we do need to figure out how > to > > change the website to build from the git checkout rather than an hg one. > > Same goes for the devguide. It would be great if we can set up web hooks > to > > immediately trigger rebuilds of those portions of the sites instead of > > having to wait until a cronjob triggers. > > > > I think we should make hg.python.org read-only but keep it around and > in sync with the GitHub repo (either via cronjobs or hooks). This > will allow people to contribute using HG in the same way that the > current GitHub clone allows people to contribute using git. It will > also avoid breaking all the tools that currently use hg.python.org > (and buys us more time to port them if/when needed). > That's easy to say, but someone also has to maintain hg.python.org then and we are doing this move partially to try and cut down on the amount of custom infrastructure that we maintain. If people are that worried about others being so adverse to using GitHub that they won't even do an anonymous clone from their servers then we can get a Bitbucket or GitLab clone set up, but I would rather try and cut out our repo hosting services if possible (who knows, maybe we can even finally retire svn.python.org thanks to shallow clones or something). > > > CPython requirements > > ================= > > There are six things to work out before we move over cpython. First, do > we > > want to split out Python 2 branches into their own repo? There might be a > > clone size benefit which obviously is nice for people on slow Internet > > connections. It also clearly separates out Python 2 from 3 and lets those > > who prefer to focus on one compared to the other do that more easily. It > > does potentially make any single fix that spans 2 and 3 require a bit > more > > work since it won't be an intra-clone change. We could also contemplate > > sub-repos for things like the Doc/ or Tools/ directories (although I > doubt > > it's worth it). > > > > I think we should keep 2/3 together. We could split the stdlib from > the rest, but that's a separate issue. > This seems to be the general consensus, so we will plan to keep cpython as a single repo. > > > Second, do we want all fixes to go into master and then cherry-pick into > > other branches, or do we want to follow our current practice of going > into > > the active feature branch and then merge into master? I personally prefer > > the former and I think most everyone else does as well, but I thought it > > should be at least thought about. > > > > Master first and cherry-picking for older branches sounds good to me, > but I don't know if switching model will have any implications, > especially while going through the history or using tools like bisect. > This seems to be the general consensus, so we will plan to cherry pick commits into older branches. > > > Third, how to handle Misc/NEWS? We can add a NEWS field to > bugs.python.org > > and then generate the NEWS file by what issues are tied to what version > and > > when they were closed. The other approach is to create a file per NEWS > entry > > in a version-specific directory (Larry created code for hg already for > this > > to give people an idea: http://bugs.python.org/issue18967). Then when > we cut > > a release we run a tool the slurps up all of the relevant files -- which > > includes files in the directory for the next feature release which > represent > > fixes which were cherry picked -- and generates the NEWS file for the > final > > release. The per-file approach is bot-friendly and also CLI-friendly, but > > potentially requires more tooling and I don't know if people feel news > > entries should be tied to the issue or in the repo (although that assumes > > people find tweaking Roundup easy :). > > > > Fourth, we need to decide exactly what commands we expect core devs to > run > > initially for committing code. Since we agreed to a linear history we > need > > to document exactly what we expect people to do for a PR to get it into > > their git repo. This will go into the devguide -- probably will want to > > start a github branch at some point -- and will act as the commands the > bot > > will want to work off of. > > > > I would like to see a complete list of steps from starting to work on > an issue to having it in the repo, at least to understand the new > workflow. This doesn't have to include all the specific commands, but > at least the basic steps (e.g. after I made a patch to I commit it and > send a pull request to the main repo, or do I push it to my GitHub > clone and push a button to send the PR? Do I need to create a branch > before I start working on an issue? > There will be a step-by-step guide in the devguide to answer all of this before we make any switch. > > > Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we > > flatten them, I believe the PR creators will get credit in the commit as > the > > author while the core dev committing will be flagged as the person doing > the > > merge (someone correct me if I'm wrong because if I am this whole point > is > > silly). With the commits containing credit directly, we can either > > automatically generate Misc/ACKS like the NEWS file or simply drop it for > > future contributors and just leave the file for past contributors since > git > > will have kept track for us. > > > > We could keep updating for regular patches with no related PR and add > a note about all the other GIT contributors (possibly with a git > command that lists all authors). > Later on we might decide to have a script that automatically adds all > the GIT contributors automatically. > This seems to be the general consensus, so we will keep Misc/ACKS around and have a tool that updates it based on git PR commits at release-time. > > > Six, we will need to update our Buildbot fleet. > > > > If we keep hg.p.o around and updated, we might not have to do this now > (even though now is better than never). > > > This gets us to the bare minimum needed to function. > > > > Parity with hg.python.org > > ---------------------------------- > > For parity, there are some Roundup integrations that will be necessary, > like > > auto-generating links, posting commits to #python-dev on IRC, etc. I'm > not > > sure if people want to block until that is all in place or not. I do > think > > we should make sure there is some web hook that can take an issue # from > the > > title of a PR and automatically posts to the corresponding issue on > > bugs.python.org that the PR exists. If people disagree then feel free > to say > > so. > > > > FWIW I started adding notes to > https://wiki.python.org/moin/TrackerDevelopmentPlanning to track > everything that needs to be done on the Roundup side. > If you prefer I can later move this to the new PEP, but for now I'm > using it to keep track of all the things that come up in the various > threads. > Nope, the wiki is fine for that sort of thing. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Jan 5 13:06:55 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 5 Jan 2016 19:06:55 +0100 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> Message-ID: <568C063F.7050608@egenix.com> On 05.01.2016 18:50, Brett Cannon wrote: > On Tue, 5 Jan 2016 at 03:22 M.-A. Lemburg wrote: > >> On 05.01.2016 06:53, Ezio Melotti wrote: >>>> Or is there some prepackaged service that >>>> we can use that will keep track of this which would cause us to not use >>>> Roundup (which might be easier, but depending on the service require >>>> everyone to re-sign)? There's also the issue of supporting people who >> want >>>> to submit code by uploading a patch to bugs.python.org but not use >> GitHub. >>>> Either way I don't want to have to ask everyone who submits a PR what >> their >>>> bugs.python.org username is and then go check that manually. >>>> >>> >>> This also brings up another problem. >>> Since the discussion about an issue happens on b.p.o and the PRs are >>> submitted on GitHub, this means that: >>> 1) users with only a GitHub account have to create a b.p.o account if >>> they want to comment on the issue (exclusing review comments); >>> 2) users with only a b.p.o account have to create a GitHub account if >>> they want to review a PR; >>> 3) users with both can comment on b.p.o and review on GitHub, but they >>> might need to login twice. >>> >>> It would be better if users didn't need to create and use two separate >> accounts. >> >> Given that we want to make it possible to move away from Github >> without too much fuzz, wouldn't it be better to have the >> PR discussions on b.p.o and Rietvield ? >> > > One of the motivating factors of this move is to get off of our fork of > Rietveld, so that's not feasible. > > >> >> If we start using Github for this, we'd lose that part of the >> review history when moving away. >> > > GitHub's API allows for exporting all of the data. > > >> >> Moving from the SF issue tracker to b.p.o was a major piece of work >> (mostly done by Martin von L?wis IIRC) and it's not clear how we >> could retrofit those discussions into the b.p.o issue discussions. >> >> Perhaps we could gateway the emails that Github sends for PR >> discussions back into b.p.o in some way (the emails contain the >> PR number, repo name and Github account name of the sender >> in the headers). >> > > I believe GitLab has a GitHub -> GitLab migration tool that keeps PR > history, so this isn't quite the issue that it initially appears to be. If that's the case, then it's fine. > If people are that worried, we could do a daily dump of the data. This would be a good idea in general. Backups are always good to have :-) > But > unless we can have all GitHub-related comments to an issue not trigger an > email update I don't think that's feasible as it would mean two emails for > every PR comment; one from GitHub and one from b.p.o. True. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jan 05 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From brett at python.org Tue Jan 5 13:13:19 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 18:13:19 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: Day 1 summary ============ Decisions made ----------------------- - We will create a python-dev team in the python org and add that team to all of the relevant repos that get migrated - We will add a GitHub field to Roundup and then write/get a CLA bot that will query Roundup to check if someone has (not) signed the CLA - For ancillary repos, the merge-of-least-resistance workflow is fine, so no need to worry about maintaining a linear history - We are going to keep all of the cpython history in a single repo - We will have PRs go against master and then we will cherry pick into bugfix branches - Misc/ACKS will be kept and we will write code to update it on occasion -- probably at release time -- based on git commit data - Seems like our current commit ID -> URL service can be updated to handle our transition Open issues ------------------- - What tools and commands will we use to convert the repos? - How do we want to handle Misc/NEWS? - What are the exact commands we are going to expect core devs to run (which will be documented in the devguide)? - Who to use for CI (although Donald is the only one to speak up and likes Travis as do I, so this might be decided)? - Who to use for code coverage (Donald has suggested codecov)? - Do we want to add GitHub login support for b.p.o to ease in adding people's GitHub username and to minimize the need for two accounts to propose a patch that has a corresponding issue? On Mon, 4 Jan 2016 at 16:42 Brett Cannon wrote: > So consider this the starting discussion of the PEP that will be the > hg.python.org -> GitHub transition PEP that I will be in charge of. Once > we have general agreement on the steps necessary I will start the actual > PEP and check it in, but I figure there's no point in have a skeleton PEP > if we can't agree on the skeleton. :) While I list steps influencing all > the repos, I want to focus on the ones stopping any repo from moving over > for now, expanding what we worry about to the cpython repo as we knock > blockers down until we move everything over and start adding GitHub perks. > > The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, > devguide, and cpython. I also think that's essentially the order we should > migrate them over. Some things will need to be universally handled before > we transition a single repo, while other things are only a blocker for some > of the repos. > > Universal blockers > ============== > There are four blockers that must be resolved before we even consider > moving a repo over. They can be solved in parallel, but they all need to > have a selected solution before we can move even the devinabox repo. > > First, we need to decide how we are going to handle adding all the core > devs to GitHub. Are we simply going to add all of them to the python > organization, or do we want something like a specific python-dev gteamroup > that gets added to all of the relevant repos? Basically I'm not sure how > picky we want to be about the people who already have organization access > on GitHub about them implicitly getting access to the cpython repo at the > end of the day (and the same goes for any of the other repos in the python > organization). For tracking names, I figure we will create a file in the > devguide where people can check in their GitHub usernames and I can > manually add people as people add themselves to the file. > > Second, CLA enforcement. As of right now people go to > https://www.python.org/psf/contrib/contrib-form/, fill in the form, and > then Ewa gets an email where she manually flips a flag in Roundup. If we > want to use a web hook to verify someone has signed a CLA then we need to > decide where the ground truth for CLAs are. Do we want to keep using > Roundup to manage CLA agreements and thus add a GitHub field in > bugs.python.org for people's profile and a web hook or bot that will > signal if someone has the flag flipped on bugs.python.org? Or is there > some prepackaged service that we can use that will keep track of this which > would cause us to not use Roundup (which might be easier, but depending on > the service require everyone to re-sign)? There's also the issue of > supporting people who want to submit code by uploading a patch to > bugs.python.org but not use GitHub. Either way I don't want to have to > ask everyone who submits a PR what their bugs.python.org username is and > then go check that manually. > > Third, how do we want to do the repo conversions? We need to choose the > tool(s) and command(s) that we want to use. There was mention of wanting a > mapping from hg commit ID to git commit ID. If we have that we could have a > static bugs.python.org/commit/ page that had the mapping embedded in > some JavaScript and if matched then we just forward them to the > corresponding GitHub commit page, otherwise just blindly forward to GitHub > and assume the ID is git-only, giving us a stable URL for commit web views. > > Fourth, for the ancillary repos of devinabox, peps, benchmarks, and > devguide, do we care if we use the GitHub merge button for PRs or do we > want to enforce a linear history with all repos? We just need to decide if > care about linear histories and then we can move forward since any bot we > create won't block us from using GitHub. > > Those four things are enough to move devinabox over. It probably is enough > for the benchmarks suite, but I have an email to speed@ asking if people > want to use this opportunity to re-evaluate the benchmark suite and make > any changes that will affect repo size (e.g., use pip to pull in the > libraries and frameworks used by a benchmark rather than vendoring their > code, making the repo much smaller). > > Website-related stuff > ================ > This also almost gets us the peps repo, but we do need to figure out how > to change the website to build from the git checkout rather than an hg one. > Same goes for the devguide. It would be great if we can set up web hooks to > immediately trigger rebuilds of those portions of the sites instead of > having to wait until a cronjob triggers. > > CPython requirements > ================= > There are six things to work out before we move over cpython. First, do we > want to split out Python 2 branches into their own repo? There might be a > clone size benefit which obviously is nice for people on slow Internet > connections. It also clearly separates out Python 2 from 3 and lets those > who prefer to focus on one compared to the other do that more easily. It > does potentially make any single fix that spans 2 and 3 require a bit more > work since it won't be an intra-clone change. We could also contemplate > sub-repos for things like the Doc/ or Tools/ directories (although I doubt > it's worth it). > > Second, do we want all fixes to go into master and then cherry-pick into > other branches, or do we want to follow our current practice of going into > the active feature branch and then merge into master? I personally prefer > the former and I think most everyone else does as well, but I thought it > should be at least thought about. > > Third, how to handle Misc/NEWS? We can add a NEWS field to bugs.python.org > and then generate the NEWS file by what issues are tied to what version and > when they were closed. The other approach is to create a file per NEWS > entry in a version-specific directory (Larry created code for hg already > for this to give people an idea: http://bugs.python.org/issue18967). Then > when we cut a release we run a tool the slurps up all of the relevant files > -- which includes files in the directory for the next feature release which > represent fixes which were cherry picked -- and generates the NEWS file for > the final release. The per-file approach is bot-friendly and also > CLI-friendly, but potentially requires more tooling and I don't know if > people feel news entries should be tied to the issue or in the repo > (although that assumes people find tweaking Roundup easy :). > > Fourth, we need to decide exactly what commands we expect core devs to run > initially for committing code. Since we agreed to a linear history we need > to document exactly what we expect people to do for a PR to get it into > their git repo. This will go into the devguide -- probably will want to > start a github branch at some point -- and will act as the commands the bot > will want to work off of. > > Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we > flatten them, I believe the PR creators will get credit in the commit as > the author while the core dev committing will be flagged as the person > doing the merge (someone correct me if I'm wrong because if I am this whole > point is silly). With the commits containing credit directly, we can either > automatically generate Misc/ACKS like the NEWS file or simply drop it for > future contributors and just leave the file for past contributors since git > will have kept track for us. > > Six, we will need to update our Buildbot fleet. > > This gets us to the bare minimum needed to function. > > Parity with hg.python.org > ---------------------------------- > For parity, there are some Roundup integrations that will be necessary, > like auto-generating links, posting commits to #python-dev on IRC, etc. I'm > not sure if people want to block until that is all in place or not. I do > think we should make sure there is some web hook that can take an issue # > from the title of a PR and automatically posts to the corresponding issue > on bugs.python.org that the PR exists. If people disagree then feel free > to say so. > > Adding perks > ========== > Now we get to some added stuff that we never had on our own > infrastructure. :) > > We should wire up CI for all PRs. I don't know if we want to go with > Travis, Codeship, or what CI provider, but we should definitely hook it up > and fully utilize the resource. This could even include running doctest > over the docs, making sure the reST markup is accurate, etc. > > Do we need to set up a web hook to trigger website rebuilds? We should at > least have a mirror on Read the Docs that is triggered by web hook so that > we have a backup of the documentation (if possible; not sure how custom our > Sphinx setup is compared to what they require to work). > > We should try to get test coverage wired up as well per CI. I don't know > if coveralls.io or some other provider is best, but we should see what is > available and find out if we can use them to either get basic coverage or > thorough coverage (read > https://hg.python.org/devinabox/file/tip/README#l124 to see what thorough > coverage entails, but it does require a checkout of coverage.py). > > We should build a bot. It must use a Monty Python reference to trigger > (black-knight, none-shall-pass, man-from-scene-24, five-questions, > what-is-your-quest, what-is-your-favourite-colour, etc.; obviously I'm > leaning towards the Black Knight or Bridge of Death scenes from the Holy > Grail for inspiration since they deal with blocking you from doing > something). It should handle specifying the commit message, what branches > to commit to/cherry pick into, and a NEWS entry (if necessary). I don't > know if it needs to do anything else as a requirement. It should probably > implement a commit queue like Zuul or Homu (and both of those can be > considered as the basis of the bot). Also gating commits on passing a test > run probably would also be good. > > I'm sure we will want to use some labels and milestones to track what PRs > are for what versions, if they are blocked on something, etc. > > --- > > Long email! :) I think that is my current brain dumped in email form. As I > said at the beginning, I think we should focus on what is blocking the > easiest repos first and then just keep knocking down blockers as we try to > move over more repos. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jan 5 13:11:11 2016 From: donald at stufft.io (Donald Stufft) Date: Tue, 5 Jan 2016 13:11:11 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: > On Jan 5, 2016, at 1:03 PM, Brett Cannon wrote: > > > > On Mon, 4 Jan 2016 at 21:54 Ezio Melotti > wrote: > On Tue, Jan 5, 2016 at 2:42 AM, Brett Cannon > wrote: > > So consider this the starting discussion of the PEP that will be the > > hg.python.org -> GitHub transition PEP that I will be in charge of. Once we > > have general agreement on the steps necessary I will start the actual PEP > > and check it in, but I figure there's no point in have a skeleton PEP if we > > can't agree on the skeleton. :) While I list steps influencing all the > > repos, I want to focus on the ones stopping any repo from moving over for > > now, expanding what we worry about to the cpython repo as we knock blockers > > down until we move everything over and start adding GitHub perks. > > > > The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, > > devguide, and cpython. > > On top of this, there is also the test repo > (https://hg.python.org/test ) and all the tracker repos > (https://hg.python.org/tracker/ ). > I think it would be useful to port the former since it will provide a > place for devs to try things out and experiment (a new test repo could > also be created though). > It would be nice to port the tracker repos too and be consistent with > the others, but it's not a priority. When we switched to HG they kept > being on SVN until I ported them, so I guess the same thing can be > done (unless R. David or Martin prefer to stick to HG). > > > I also think that's essentially the order we should > > migrate them over. Some things will need to be universally handled before we > > transition a single repo, while other things are only a blocker for some of > > the repos. > > > > Universal blockers > > ============== > > There are four blockers that must be resolved before we even consider moving > > a repo over. They can be solved in parallel, but they all need to have a > > selected solution before we can move even the devinabox repo. > > > > First, we need to decide how we are going to handle adding all the core devs > > to GitHub. Are we simply going to add all of them to the python > > organization, or do we want something like a specific python-dev gteamroup > > that gets added to all of the relevant repos? Basically I'm not sure how > > picky we want to be about the people who already have organization access on > > GitHub about them implicitly getting access to the cpython repo at the end > > of the day (and the same goes for any of the other repos in the python > > organization). For tracking names, I figure we will create a file in the > > devguide where people can check in their GitHub usernames and I can manually > > add people as people add themselves to the file. > > > > I think the current list of core-devs should be converted to a group > and given access to the same repos they have access to now (i.e. > cpython/devguide/peps and possibly others). Then additional > repo-specific groups can be created in case we want to let specific > contributors work on peps or the devguide. > > This seems to be the general consensus, so we will create a python-dev team under the python org and add the core devs there. Just to expand on this more, GitHub has recently updated their permission model more to also allow ?outside contributors? to be added to a single repository without adding them to a group (Not sure if this would be useful for us or not). I imagine the python-dev team would have ?Write Access? (See https://help.github.com/articles/repository-permission-levels-for-an-organization/ ) unless we think it needs admin access. > > > > Second, CLA enforcement. As of right now people go to > > https://www.python.org/psf/contrib/contrib-form/ , fill in the form, and then > > Ewa gets an email where she manually flips a flag in Roundup. If we want to > > use a web hook to verify someone has signed a CLA then we need to decide > > where the ground truth for CLAs are. Do we want to keep using Roundup to > > manage CLA agreements and thus add a GitHub field in bugs.python.org for > > people's profile and a web hook or bot that will signal if someone has the > > flag flipped on bugs.python.org ? > > This can be done. We can add a "GitHub" username field to Roundup > users so that we can link the two. > > OK, so it sounds like we will stick with our current CLA signing flow and write our own CLA bot that will query Roundup as to whether someone has signed the CLA or not and then throw up a banner signalling if someone has (not) signed and an appropriate link to the CLA. That will require some Roundup work and the creation of the bot. We can set a commit status that will show red if the user hasn?t signed the CLA (just like if Travis tests failed or so). No need to use a banner or anything. > > I should also mention, any bot creations we do should abstract out the code review tool so that when we change providers again in the future it will be more straight-forward to just update some select APIs rather than rewrite every bot we create. > > > > > Or is there some prepackaged service that > > we can use that will keep track of this which would cause us to not use > > Roundup (which might be easier, but depending on the service require > > everyone to re-sign)? There's also the issue of supporting people who want > > to submit code by uploading a patch to bugs.python.org but not use GitHub. > > Either way I don't want to have to ask everyone who submits a PR what their > > bugs.python.org username is and then go check that manually. > > > > This also brings up another problem. > Since the discussion about an issue happens on b.p.o and the PRs are > submitted on GitHub, this means that: > 1) users with only a GitHub account have to create a b.p.o account if > they want to comment on the issue (exclusing review comments); > 2) users with only a b.p.o account have to create a GitHub account if > they want to review a PR; > 3) users with both can comment on b.p.o and review on GitHub, but they > might need to login twice. > > It would be better if users didn't need to create and use two separate accounts. > > If we can add GitHub as a login/creation option for b.p.o accounts then that solves that. But I'm willing to bet a majority of people will already have a GitHub account and we have always required the b.p.o account so #1 is the going to be the common case. > > > > Third, how do we want to do the repo conversions? We need to choose the > > tool(s) and command(s) that we want to use. There was mention of wanting a > > mapping from hg commit ID to git commit ID. If we have that we could have a > > static bugs.python.org/commit/ page that had the mapping embedded in > > some JavaScript and if matched then we just forward them to the > > corresponding GitHub commit page, otherwise just blindly forward to GitHub > > and assume the ID is git-only, giving us a stable URL for commit web views. > > > > As I mentioned on python-committers, we already have > https://hg.python.org/lookup/ . > This is currently used to map SVN->HG (e.g. > https://hg.python.org/lookup/r12345 ), and should be extended to > handle cs ids too. > The b.p.o linkifier can just convert all revision numbers and cs ids > to a https://hg.python.org/lookup/ link and let the lookup page figure > out where to redirect the user. > > > Fourth, for the ancillary repos of devinabox, peps, benchmarks, and > > devguide, do we care if we use the GitHub merge button for PRs or do we want > > to enforce a linear history with all repos? We just need to decide if care > > about linear histories and then we can move forward since any bot we create > > won't block us from using GitHub. > > > > Those four things are enough to move devinabox over. It probably is enough > > for the benchmarks suite, but I have an email to speed@ asking if people > > want to use this opportunity to re-evaluate the benchmark suite and make any > > changes that will affect repo size (e.g., use pip to pull in the libraries > > and frameworks used by a benchmark rather than vendoring their code, making > > the repo much smaller). > > > > Website-related stuff > > ================ > > This also almost gets us the peps repo, but we do need to figure out how to > > change the website to build from the git checkout rather than an hg one. > > Same goes for the devguide. It would be great if we can set up web hooks to > > immediately trigger rebuilds of those portions of the sites instead of > > having to wait until a cronjob triggers. > > > > I think we should make hg.python.org read-only but keep it around and > in sync with the GitHub repo (either via cronjobs or hooks). This > will allow people to contribute using HG in the same way that the > current GitHub clone allows people to contribute using git. It will > also avoid breaking all the tools that currently use hg.python.org > (and buys us more time to port them if/when needed). > > That's easy to say, but someone also has to maintain hg.python.org then and we are doing this move partially to try and cut down on the amount of custom infrastructure that we maintain. If people are that worried about others being so adverse to using GitHub that they won't even do an anonymous clone from their servers then we can get a Bitbucket or GitLab clone set up, but I would rather try and cut out our repo hosting services if possible (who knows, maybe we can even finally retire svn.python.org thanks to shallow clones or something). I think a mirror is a different level of involvement than running the main repository. We won?t have to worry about backups, if the server goes down people can still contribute, etc. Most likely it will also be able to be resized to a smaller server as well (but maybe not). I don?t think any of the infra team would be upset at one less server to manage though, so I?m more than happy either way. > > > > CPython requirements > > ================= > > There are six things to work out before we move over cpython. First, do we > > want to split out Python 2 branches into their own repo? There might be a > > clone size benefit which obviously is nice for people on slow Internet > > connections. It also clearly separates out Python 2 from 3 and lets those > > who prefer to focus on one compared to the other do that more easily. It > > does potentially make any single fix that spans 2 and 3 require a bit more > > work since it won't be an intra-clone change. We could also contemplate > > sub-repos for things like the Doc/ or Tools/ directories (although I doubt > > it's worth it). > > > > I think we should keep 2/3 together. We could split the stdlib from > the rest, but that's a separate issue. > > This seems to be the general consensus, so we will plan to keep cpython as a single repo. > > > > Second, do we want all fixes to go into master and then cherry-pick into > > other branches, or do we want to follow our current practice of going into > > the active feature branch and then merge into master? I personally prefer > > the former and I think most everyone else does as well, but I thought it > > should be at least thought about. > > > > Master first and cherry-picking for older branches sounds good to me, > but I don't know if switching model will have any implications, > especially while going through the history or using tools like bisect. > > This seems to be the general consensus, so we will plan to cherry pick commits into older branches. > > > > Third, how to handle Misc/NEWS? We can add a NEWS field to bugs.python.org > > and then generate the NEWS file by what issues are tied to what version and > > when they were closed. The other approach is to create a file per NEWS entry > > in a version-specific directory (Larry created code for hg already for this > > to give people an idea: http://bugs.python.org/issue18967 ). Then when we cut > > a release we run a tool the slurps up all of the relevant files -- which > > includes files in the directory for the next feature release which represent > > fixes which were cherry picked -- and generates the NEWS file for the final > > release. The per-file approach is bot-friendly and also CLI-friendly, but > > potentially requires more tooling and I don't know if people feel news > > entries should be tied to the issue or in the repo (although that assumes > > people find tweaking Roundup easy :). > > > > Fourth, we need to decide exactly what commands we expect core devs to run > > initially for committing code. Since we agreed to a linear history we need > > to document exactly what we expect people to do for a PR to get it into > > their git repo. This will go into the devguide -- probably will want to > > start a github branch at some point -- and will act as the commands the bot > > will want to work off of. > > > > I would like to see a complete list of steps from starting to work on > an issue to having it in the repo, at least to understand the new > workflow. This doesn't have to include all the specific commands, but > at least the basic steps (e.g. after I made a patch to I commit it and > send a pull request to the main repo, or do I push it to my GitHub > clone and push a button to send the PR? Do I need to create a branch > before I start working on an issue? > > There will be a step-by-step guide in the devguide to answer all of this before we make any switch. > > > > Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we > > flatten them, I believe the PR creators will get credit in the commit as the > > author while the core dev committing will be flagged as the person doing the > > merge (someone correct me if I'm wrong because if I am this whole point is > > silly). With the commits containing credit directly, we can either > > automatically generate Misc/ACKS like the NEWS file or simply drop it for > > future contributors and just leave the file for past contributors since git > > will have kept track for us. > > > > We could keep updating for regular patches with no related PR and add > a note about all the other GIT contributors (possibly with a git > command that lists all authors). > Later on we might decide to have a script that automatically adds all > the GIT contributors automatically. > > This seems to be the general consensus, so we will keep Misc/ACKS around and have a tool that updates it based on git PR commits at release-time. > > > > Six, we will need to update our Buildbot fleet. > > > > If we keep hg.p.o around and updated, we might not have to do this now > (even though now is better than never). > > > This gets us to the bare minimum needed to function. > > > > Parity with hg.python.org > > ---------------------------------- > > For parity, there are some Roundup integrations that will be necessary, like > > auto-generating links, posting commits to #python-dev on IRC, etc. I'm not > > sure if people want to block until that is all in place or not. I do think > > we should make sure there is some web hook that can take an issue # from the > > title of a PR and automatically posts to the corresponding issue on > > bugs.python.org that the PR exists. If people disagree then feel free to say > > so. > > > > FWIW I started adding notes to > https://wiki.python.org/moin/TrackerDevelopmentPlanning to track > everything that needs to be done on the Roundup side. > If you prefer I can later move this to the new PEP, but for now I'm > using it to keep track of all the things that come up in the various > threads. > > Nope, the wiki is fine for that sort of thing. > > -Brett > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Tue Jan 5 13:25:54 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 18:25:54 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, 5 Jan 2016 at 10:11 Donald Stufft wrote: > On Jan 5, 2016, at 1:03 PM, Brett Cannon wrote: > > > > On Mon, 4 Jan 2016 at 21:54 Ezio Melotti wrote: > >> On Tue, Jan 5, 2016 at 2:42 AM, Brett Cannon wrote: >> > So consider this the starting discussion of the PEP that will be the >> > hg.python.org -> GitHub transition PEP that I will be in charge of. >> Once we >> > have general agreement on the steps necessary I will start the actual >> PEP >> > and check it in, but I figure there's no point in have a skeleton PEP >> if we >> > can't agree on the skeleton. :) While I list steps influencing all the >> > repos, I want to focus on the ones stopping any repo from moving over >> for >> > now, expanding what we worry about to the cpython repo as we knock >> blockers >> > down until we move everything over and start adding GitHub perks. >> > >> > The way I see it, we have 4 repos to move: devinabox, benchmarks, peps, >> > devguide, and cpython. >> >> On top of this, there is also the test repo >> (https://hg.python.org/test) and all the tracker repos >> (https://hg.python.org/tracker/). >> I think it would be useful to port the former since it will provide a >> place for devs to try things out and experiment (a new test repo could >> also be created though). >> It would be nice to port the tracker repos too and be consistent with >> the others, but it's not a priority. When we switched to HG they kept >> being on SVN until I ported them, so I guess the same thing can be >> done (unless R. David or Martin prefer to stick to HG). >> >> > I also think that's essentially the order we should >> > migrate them over. Some things will need to be universally handled >> before we >> > transition a single repo, while other things are only a blocker for >> some of >> > the repos. >> > >> > Universal blockers >> > ============== >> > There are four blockers that must be resolved before we even consider >> moving >> > a repo over. They can be solved in parallel, but they all need to have a >> > selected solution before we can move even the devinabox repo. >> > >> > First, we need to decide how we are going to handle adding all the core >> devs >> > to GitHub. Are we simply going to add all of them to the python >> > organization, or do we want something like a specific python-dev >> gteamroup >> > that gets added to all of the relevant repos? Basically I'm not sure how >> > picky we want to be about the people who already have organization >> access on >> > GitHub about them implicitly getting access to the cpython repo at the >> end >> > of the day (and the same goes for any of the other repos in the python >> > organization). For tracking names, I figure we will create a file in the >> > devguide where people can check in their GitHub usernames and I can >> manually >> > add people as people add themselves to the file. >> > >> >> I think the current list of core-devs should be converted to a group >> and given access to the same repos they have access to now (i.e. >> cpython/devguide/peps and possibly others). Then additional >> repo-specific groups can be created in case we want to let specific >> contributors work on peps or the devguide. >> > > This seems to be the general consensus, so we will create a python-dev > team under the python org and add the core devs there. > > > Just to expand on this more, GitHub has recently updated their permission > model more to also allow ?outside contributors? to be added to a single > repository without adding them to a group (Not sure if this would be useful > for us or not). > It will be on the ancillary repos like peps and devguide where we may want to be more liberal in letting people commit (e.g., maybe we want to add all PEP authors as outside contributors so that they can maintain their own PEP). > > I imagine the python-dev team would have ?Write Access? (See > https://help.github.com/articles/repository-permission-levels-for-an-organization/) > unless we think it needs admin access. > I think write access is perfect for the vast majority of people. The question becomes if we want to add anyone specifically with admin access to help manage repo stuff like web hooks, or if we just bug someone on the infrastructure team to set it for us. I do think that some of us -- probably those of us on hgaccounts -- should be team owners. > > > >> >> > Second, CLA enforcement. As of right now people go to >> > https://www.python.org/psf/contrib/contrib-form/, fill in the form, >> and then >> > Ewa gets an email where she manually flips a flag in Roundup. If we >> want to >> > use a web hook to verify someone has signed a CLA then we need to decide >> > where the ground truth for CLAs are. Do we want to keep using Roundup to >> > manage CLA agreements and thus add a GitHub field in bugs.python.org >> for >> > people's profile and a web hook or bot that will signal if someone has >> the >> > flag flipped on bugs.python.org? >> >> This can be done. We can add a "GitHub" username field to Roundup >> users so that we can link the two. >> > > OK, so it sounds like we will stick with our current CLA signing flow and > write our own CLA bot that will query Roundup as to whether someone has > signed the CLA or not and then throw up a banner signalling if someone has > (not) signed and an appropriate link to the CLA. That will require some > Roundup work and the creation of the bot. > > > We can set a commit status that will show red if the user hasn?t signed > the CLA (just like if Travis tests failed or so). No need to use a banner > or anything. > > > I should also mention, any bot creations we do should abstract out the > code review tool so that when we change providers again in the future it > will be more straight-forward to just update some select APIs rather than > rewrite every bot we create. > > >> >> >> > Or is there some prepackaged service that >> > we can use that will keep track of this which would cause us to not use >> > Roundup (which might be easier, but depending on the service require >> > everyone to re-sign)? There's also the issue of supporting people who >> want >> > to submit code by uploading a patch to bugs.python.org but not use >> GitHub. >> > Either way I don't want to have to ask everyone who submits a PR what >> their >> > bugs.python.org username is and then go check that manually. >> > >> >> This also brings up another problem. >> Since the discussion about an issue happens on b.p.o and the PRs are >> submitted on GitHub, this means that: >> 1) users with only a GitHub account have to create a b.p.o account if >> they want to comment on the issue (exclusing review comments); >> 2) users with only a b.p.o account have to create a GitHub account if >> they want to review a PR; >> 3) users with both can comment on b.p.o and review on GitHub, but they >> might need to login twice. >> >> It would be better if users didn't need to create and use two separate >> accounts. >> > > If we can add GitHub as a login/creation option for b.p.o accounts then > that solves that. But I'm willing to bet a majority of people will already > have a GitHub account and we have always required the b.p.o account so #1 > is the going to be the common case. > > >> >> > Third, how do we want to do the repo conversions? We need to choose the >> > tool(s) and command(s) that we want to use. There was mention of >> wanting a >> > mapping from hg commit ID to git commit ID. If we have that we could >> have a >> > static bugs.python.org/commit/ page that had the mapping embedded >> in >> > some JavaScript and if matched then we just forward them to the >> > corresponding GitHub commit page, otherwise just blindly forward to >> GitHub >> > and assume the ID is git-only, giving us a stable URL for commit web >> views. >> > >> >> As I mentioned on python-committers, we already have >> https://hg.python.org/lookup/ . >> This is currently used to map SVN->HG (e.g. >> https://hg.python.org/lookup/r12345 ), and should be extended to >> handle cs ids too. >> The b.p.o linkifier can just convert all revision numbers and cs ids >> to a https://hg.python.org/lookup/ link and let the lookup page figure >> out where to redirect the user. >> >> > Fourth, for the ancillary repos of devinabox, peps, benchmarks, and >> > devguide, do we care if we use the GitHub merge button for PRs or do we >> want >> > to enforce a linear history with all repos? We just need to decide if >> care >> > about linear histories and then we can move forward since any bot we >> create >> > won't block us from using GitHub. >> > >> > Those four things are enough to move devinabox over. It probably is >> enough >> > for the benchmarks suite, but I have an email to speed@ asking if >> people >> > want to use this opportunity to re-evaluate the benchmark suite and >> make any >> > changes that will affect repo size (e.g., use pip to pull in the >> libraries >> > and frameworks used by a benchmark rather than vendoring their code, >> making >> > the repo much smaller). >> > >> > Website-related stuff >> > ================ >> > This also almost gets us the peps repo, but we do need to figure out >> how to >> > change the website to build from the git checkout rather than an hg one. >> > Same goes for the devguide. It would be great if we can set up web >> hooks to >> > immediately trigger rebuilds of those portions of the sites instead of >> > having to wait until a cronjob triggers. >> > >> >> I think we should make hg.python.org read-only but keep it around and >> in sync with the GitHub repo (either via cronjobs or hooks). This >> will allow people to contribute using HG in the same way that the >> current GitHub clone allows people to contribute using git. It will >> also avoid breaking all the tools that currently use hg.python.org >> (and buys us more time to port them if/when needed). >> > > That's easy to say, but someone also has to maintain hg.python.org then > and we are doing this move partially to try and cut down on the amount of > custom infrastructure that we maintain. If people are that worried about > others being so adverse to using GitHub that they won't even do an > anonymous clone from their servers then we can get a Bitbucket or GitLab > clone set up, but I would rather try and cut out our repo hosting services > if possible (who knows, maybe we can even finally retire svn.python.org thanks > to shallow clones or something). > > > I think a mirror is a different level of involvement than running the main > repository. We won?t have to worry about backups, if the server goes down > people can still contribute, etc. Most likely it will also be able to be > resized to a smaller server as well (but maybe not). I don?t think any of > the infra team would be upset at one less server to manage though, so I?m > more than happy either way. > Then I leave it to the infrastructure team to discuss amongst themselves to tell me if they think keeping hg.python.org up as a mirror is no big deal or if they simply want to set up some mirror at another host like Bitbucket or GitLab. Since it's your burden in the end it should be your decision as to whether you want to shoulder it to give people a purely open source way to contribute to Python (I also leave it in your hands whether making it a git or hg mirror is easier). -Brett > > > >> >> > CPython requirements >> > ================= >> > There are six things to work out before we move over cpython. First, do >> we >> > want to split out Python 2 branches into their own repo? There might be >> a >> > clone size benefit which obviously is nice for people on slow Internet >> > connections. It also clearly separates out Python 2 from 3 and lets >> those >> > who prefer to focus on one compared to the other do that more easily. It >> > does potentially make any single fix that spans 2 and 3 require a bit >> more >> > work since it won't be an intra-clone change. We could also contemplate >> > sub-repos for things like the Doc/ or Tools/ directories (although I >> doubt >> > it's worth it). >> > >> >> I think we should keep 2/3 together. We could split the stdlib from >> the rest, but that's a separate issue. >> > > This seems to be the general consensus, so we will plan to keep cpython as > a single repo. > > >> >> > Second, do we want all fixes to go into master and then cherry-pick into >> > other branches, or do we want to follow our current practice of going >> into >> > the active feature branch and then merge into master? I personally >> prefer >> > the former and I think most everyone else does as well, but I thought it >> > should be at least thought about. >> > >> >> Master first and cherry-picking for older branches sounds good to me, >> but I don't know if switching model will have any implications, >> especially while going through the history or using tools like bisect. >> > > This seems to be the general consensus, so we will plan to cherry pick > commits into older branches. > > >> >> > Third, how to handle Misc/NEWS? We can add a NEWS field to >> bugs.python.org >> > and then generate the NEWS file by what issues are tied to what version >> and >> > when they were closed. The other approach is to create a file per NEWS >> entry >> > in a version-specific directory (Larry created code for hg already for >> this >> > to give people an idea: http://bugs.python.org/issue18967). Then when >> we cut >> > a release we run a tool the slurps up all of the relevant files -- which >> > includes files in the directory for the next feature release which >> represent >> > fixes which were cherry picked -- and generates the NEWS file for the >> final >> > release. The per-file approach is bot-friendly and also CLI-friendly, >> but >> > potentially requires more tooling and I don't know if people feel news >> > entries should be tied to the issue or in the repo (although that >> assumes >> > people find tweaking Roundup easy :). >> > >> > Fourth, we need to decide exactly what commands we expect core devs to >> run >> > initially for committing code. Since we agreed to a linear history we >> need >> > to document exactly what we expect people to do for a PR to get it into >> > their git repo. This will go into the devguide -- probably will want to >> > start a github branch at some point -- and will act as the commands the >> bot >> > will want to work off of. >> > >> >> I would like to see a complete list of steps from starting to work on >> an issue to having it in the repo, at least to understand the new >> workflow. This doesn't have to include all the specific commands, but >> at least the basic steps (e.g. after I made a patch to I commit it and >> send a pull request to the main repo, or do I push it to my GitHub >> clone and push a button to send the PR? Do I need to create a branch >> before I start working on an issue? >> > > There will be a step-by-step guide in the devguide to answer all of this > before we make any switch. > > >> >> > Fifth, what to do about Misc/ACKS? Since we are using PRs, even if we >> > flatten them, I believe the PR creators will get credit in the commit >> as the >> > author while the core dev committing will be flagged as the person >> doing the >> > merge (someone correct me if I'm wrong because if I am this whole point >> is >> > silly). With the commits containing credit directly, we can either >> > automatically generate Misc/ACKS like the NEWS file or simply drop it >> for >> > future contributors and just leave the file for past contributors since >> git >> > will have kept track for us. >> > >> >> We could keep updating for regular patches with no related PR and add >> a note about all the other GIT contributors (possibly with a git >> command that lists all authors). >> Later on we might decide to have a script that automatically adds all >> the GIT contributors automatically. >> > > This seems to be the general consensus, so we will keep Misc/ACKS around > and have a tool that updates it based on git PR commits at release-time. > > >> >> > Six, we will need to update our Buildbot fleet. >> > >> >> If we keep hg.p.o around and updated, we might not have to do this now >> (even though now is better than never). >> >> > This gets us to the bare minimum needed to function. >> > >> > Parity with hg.python.org >> > ---------------------------------- >> > For parity, there are some Roundup integrations that will be necessary, >> like >> > auto-generating links, posting commits to #python-dev on IRC, etc. I'm >> not >> > sure if people want to block until that is all in place or not. I do >> think >> > we should make sure there is some web hook that can take an issue # >> from the >> > title of a PR and automatically posts to the corresponding issue on >> > bugs.python.org that the PR exists. If people disagree then feel free >> to say >> > so. >> > >> >> FWIW I started adding notes to >> https://wiki.python.org/moin/TrackerDevelopmentPlanning to track >> everything that needs to be done on the Roundup side. >> If you prefer I can later move this to the new PEP, but for now I'm >> using it to keep track of all the things that come up in the various >> threads. >> > > Nope, the wiki is fine for that sort of thing. > > -Brett > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jan 5 13:48:30 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 05 Jan 2016 13:48:30 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> Message-ID: <20160105184832.89C59251164@webabinitio.net> On Tue, 05 Jan 2016 17:50:53 +0000, Brett Cannon wrote: > If people are that worried, we could do a daily dump of the data. But > unless we can have all GitHub-related comments to an issue not trigger an > email update I don't think that's feasible as it would mean two emails for > every PR comment; one from GitHub and one from b.p.o. We can fairly easily have all github-originated comments not trigger an email to the nosy list. We'd just need to add a check for the originating email address to the nosy reaction detector. Even nicer would be to make it a per-user setting (I'd rather get the emails from b.p.o. and disable them on github, myself), but that would be a bit more work. It would be nice to have *all* of the discussion in one place. --David From brett at python.org Tue Jan 5 14:18:10 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 19:18:10 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: <20160105184832.89C59251164@webabinitio.net> References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Tue, 5 Jan 2016 at 10:48 R. David Murray wrote: > On Tue, 05 Jan 2016 17:50:53 +0000, Brett Cannon wrote: > > If people are that worried, we could do a daily dump of the data. But > > unless we can have all GitHub-related comments to an issue not trigger an > > email update I don't think that's feasible as it would mean two emails > for > > every PR comment; one from GitHub and one from b.p.o. > > We can fairly easily have all github-originated comments not trigger an > email to the nosy list. We'd just need to add a check for the > originating email address to the nosy reaction detector. Even nicer > would be to make it a per-user setting (I'd rather get the emails from > b.p.o. and disable them on github, myself), but that would be a bit more > work. > > It would be nice to have *all* of the discussion in one place. > If someone wants to put the work in to make that happen then that's fine by me, but I do consider this a nicety and not a blocker. We haven't had code review comments go into b.p.o on Rietveld either so this isn't that far off from what we have now, especially if we block the transition on adding a link back to the GitHub PR through a web hook. -------------- next part -------------- An HTML attachment was scrubbed... URL: From senthil at uthcode.com Tue Jan 5 15:35:42 2016 From: senthil at uthcode.com (Senthil Kumaran) Date: Tue, 5 Jan 2016 12:35:42 -0800 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: Hi Brett, On Tue, Jan 5, 2016 at 10:13 AM, Brett Cannon wrote: > - We are going to keep all of the cpython history in a single repo Are we not going to have 2.7 as a separate repo? I believe, this is possible even if we keep cpython with full history (and branches). Thanks, Senthil -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 5 15:36:47 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Jan 2016 12:36:47 -0800 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, Jan 5, 2016 at 12:35 PM, Senthil Kumaran wrote: > Are we not going to have 2.7 as a separate repo? > Why would you want this? > I believe, this is possible even if we keep cpython with full history (and > branches). > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From senthil at uthcode.com Tue Jan 5 15:44:18 2016 From: senthil at uthcode.com (Senthil Kumaran) Date: Tue, 5 Jan 2016 12:44:18 -0800 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, Jan 5, 2016 at 12:36 PM, Guido van Rossum wrote: > > > Why would you want this? > In general, our changes to 2.7 are separate from 3.x lines. So the pull requests and reviews could be handled separately. We could encourage feature contributions only to cpython (3.x) repo and any maintainance improvements from the folks who have 2.7 in production could be handled in 2.7 repo separately. Also, I was mapping our hg branches to git repos. Currently we have 3.4->3.5->default (active) And 2.7(active) -- Senthil -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Jan 5 16:04:40 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 21:04:40 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, 5 Jan 2016 at 12:35 Senthil Kumaran wrote: > Hi Brett, > > On Tue, Jan 5, 2016 at 10:13 AM, Brett Cannon wrote: > >> - We are going to keep all of the cpython history in a single repo > > > Are we not going to have 2.7 as a separate repo? > I believe, this is possible even if we keep cpython with full history (and > branches). > No, we are not going to break 2.7 into a separate repo; everyone else wants to keep the cpython repo as-is with all branches in a single repo. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From berker.peksag at gmail.com Tue Jan 5 16:28:18 2016 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Tue, 5 Jan 2016 23:28:18 +0200 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, Jan 5, 2016 at 8:13 PM, Brett Cannon wrote: > Day 1 summary > ============ > > Decisions made > ----------------------- > - We will create a python-dev team in the python org and add that team to > all of the relevant repos that get migrated > - We will add a GitHub field to Roundup and then write/get a CLA bot that > will query Roundup to check if someone has (not) signed the CLA > - For ancillary repos, the merge-of-least-resistance workflow is fine, so no > need to worry about maintaining a linear history > - We are going to keep all of the cpython history in a single repo > - We will have PRs go against master and then we will cherry pick into > bugfix branches > - Misc/ACKS will be kept and we will write code to update it on occasion -- > probably at release time -- based on git commit data > - Seems like our current commit ID -> URL service can be updated to handle > our transition > > > Open issues > ------------------- > - What tools and commands will we use to convert the repos? > - How do we want to handle Misc/NEWS? > - What are the exact commands we are going to expect core devs to run (which > will be documented in the devguide)? I use the following commands for python.org: $ git pr NNN "git pr" is an alias and an equivalent of $ git fetch upstream pull/NNN/head:pr/NNN $ git checkout pr/NNN For example, if I want to review https://github.com/python/pythondotorg/pull/859, I use "git pr 859". Run tests, tweak commit message, code style, and commit $ git commit -m "Update foo bar" Squash commits $ git rebase -i HEAD~2 And merge $ git checkout master $ git merge --ff-only pr/NNN (--ff-only can be added to .gitconfig) Backport to 3.5 (the branch name is "release" for python.org) $ git checkout 3.5 $ git cherry-pick `git rev-parse master` And push to upstream $ git push --all upstream We probably should mention the -u option (or create a new section for Git configuration and best practices) in the devguide. > - Who to use for CI (although Donald is the only one to speak up and likes > Travis as do I, so this might be decided)? +1 to use Travis as a pre-merge CI. I'd vote for keeping Travis configuration simple (e.g. use only Linux and GCC, run rstlint in Doc/) --Berker From ericsnowcurrently at gmail.com Tue Jan 5 17:19:44 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 5 Jan 2016 15:19:44 -0700 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, Jan 5, 2016 at 11:13 AM, Brett Cannon wrote: > Day 1 summary > ============ > > Decisions made > ----------------------- > > Open issues > ------------------- And a couple things that we are punting on: * code review tool (if GH proves undesirable) * separate (sub)repos for docs/tutorials--they could have a less restricted workflow than the rest of the cpython repo, a la the devguide Both of these can wait until later, though they still deserve mention in the PEP. -eric From brett at python.org Tue Jan 5 17:40:27 2016 From: brett at python.org (Brett Cannon) Date: Tue, 05 Jan 2016 22:40:27 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: On Tue, 5 Jan 2016 at 14:19 Eric Snow wrote: > On Tue, Jan 5, 2016 at 11:13 AM, Brett Cannon wrote: > > Day 1 summary > > ============ > > > > Decisions made > > ----------------------- > > > > Open issues > > ------------------- > > And a couple things that we are punting on: > > * code review tool (if GH proves undesirable) > Well, that's implicit if we find GitHub doesn't work for us for code review. I don't think it requires explicitly calling it out. > * separate (sub)repos for docs/tutorials--they could have a less > restricted workflow than the rest of the cpython repo, a la the > devguide > Sure, it can be mentioned. > > Both of these can wait until later, though they still deserve mention > in the PEP. You really don't like GitHub's review tool, huh? ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.chammas at gmail.com Tue Jan 5 17:54:33 2016 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Tue, 05 Jan 2016 22:54:33 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: We can set a commit status that will show red if the user hasn?t signed the CLA (just like if Travis tests failed or so). No need to use a banner or anything. This is a great idea. Almost any automated check we want to run against PRs can be captured as a Travis/CI test that shows up on the PR with its own status . Nick ? On Tue, Jan 5, 2016 at 5:40 PM Brett Cannon wrote: > On Tue, 5 Jan 2016 at 14:19 Eric Snow wrote: > >> On Tue, Jan 5, 2016 at 11:13 AM, Brett Cannon wrote: >> > Day 1 summary >> > ============ >> > >> > Decisions made >> > ----------------------- >> > >> > Open issues >> > ------------------- >> >> And a couple things that we are punting on: >> >> * code review tool (if GH proves undesirable) >> > > Well, that's implicit if we find GitHub doesn't work for us for code > review. I don't think it requires explicitly calling it out. > > >> * separate (sub)repos for docs/tutorials--they could have a less >> restricted workflow than the rest of the cpython repo, a la the >> devguide >> > > Sure, it can be mentioned. > > >> >> Both of these can wait until later, though they still deserve mention >> in the PEP. > > > You really don't like GitHub's review tool, huh? ;) > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Jan 5 19:24:51 2016 From: barry at python.org (Barry Warsaw) Date: Tue, 5 Jan 2016 19:24:51 -0500 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition References: Message-ID: <20160105192451.36d7fea9@anarchist.wooz.org> On Jan 05, 2016, at 06:03 PM, Brett Cannon wrote: >If people are that worried about others being so adverse to using GitHub that >they won't even do an anonymous clone from their servers then we can get a >Bitbucket or GitLab clone set up Once the migration settles down, I do plan on hooking up this GitLab mirror, which was original created as a PEP 507 experiment: https://gitlab.com/python-devs/python Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From nicolas.alvarez at gmail.com Tue Jan 5 22:37:50 2016 From: nicolas.alvarez at gmail.com (=?utf-8?b?Tmljb2zDoXM=?= Alvarez) Date: Wed, 6 Jan 2016 03:37:50 +0000 (UTC) Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition References: <93574A50-8E31-49D1-80A4-590AC4C347D7@stufft.io> Message-ID: Donald Stufft writes: > > There is CLAHub (https://www.clahub.com/) but I don?t have any idea how good it is, I just know of it?s existence. CLAHub is apparently dying, see https://github.com/clahub/clahub/issues/111. Another option is https://cla-assistant.io/ which already has features that CLAHub doesn't, such as posting "You haven't signed the CLA" comments on review requests from new contributors. -- Nicol?s Apologies for crappy quoting etc. I'm using Gmane to post this message. From nicolas.alvarez at gmail.com Tue Jan 5 22:52:05 2016 From: nicolas.alvarez at gmail.com (=?UTF-8?Q?Nicol=C3=A1s_Alvarez?=) Date: Wed, 6 Jan 2016 00:52:05 -0300 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: 2016-01-05 15:13 GMT-03:00 Brett Cannon : > Open issues > ------------------- > - What tools and commands will we use to convert the repos? I will investigate this soon. I don't claim ESR-level experience in repository conversion, but I did migrate most of KDE from SVN to Git (which was one hell of a task) :) However, I will be on vacation and trying to stay away from my email inbox (except urgent KDE sysadmin matters) from Jan 9th to Jan ~15th. What's the general timeframe you're thinking for all this? I know Python is not a corporate project with "deadlines", but I bet nobody would like "CPython will be in GitHub by early 2017"... -- Nicol?s From ezio.melotti at gmail.com Wed Jan 6 02:33:58 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Wed, 6 Jan 2016 09:33:58 +0200 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Tue, Jan 5, 2016 at 9:18 PM, Brett Cannon wrote: > > > On Tue, 5 Jan 2016 at 10:48 R. David Murray wrote: >> >> On Tue, 05 Jan 2016 17:50:53 +0000, Brett Cannon wrote: >> > If people are that worried, we could do a daily dump of the data. But >> > unless we can have all GitHub-related comments to an issue not trigger >> > an >> > email update I don't think that's feasible as it would mean two emails >> > for >> > every PR comment; one from GitHub and one from b.p.o. >> >> We can fairly easily have all github-originated comments not trigger an >> email to the nosy list. We'd just need to add a check for the >> originating email address to the nosy reaction detector. Even nicer >> would be to make it a per-user setting (I'd rather get the emails from >> b.p.o. and disable them on github, myself), but that would be a bit more >> work. >> >> It would be nice to have *all* of the discussion in one place. > > > If someone wants to put the work in to make that happen then that's fine by > me, but I do consider this a nicety and not a blocker. We haven't had code > review comments go into b.p.o on Rietveld either so this isn't that far off > from what we have now, especially if we block the transition on adding a > link back to the GitHub PR through a web hook. I agree it's not a blocker, but a better integration between Rietveld and b.p.o was often requested (I tried -- and failed -- to tackle the issue a couple of times). What I want to avoid is having people on github missing b.p.o messages, and people on b.p.o missing PRs/reviews on github. (This was already happening when people were discussing the patch on Rietveld, and others were only looking at b.p.o and/or missing the reviews, both because the email from rietveld ended up in the spam folder, and because there was no way to know from b.p.o if someone posted a review.) I think it's ok to have discussions on b.p.o and reviews on github (similarly to what we currently do with Rietveld), as long as the everyone gets notified. Discussions that are not strictly related to the code/review shouldn't happen on github. As for the notifications, I think the best option is: * when a PR is posted, it's also automatically added to the issue (to the list of PRs, similar to the list of patches; generates an email too); * when a review is posted, a new message with a link is added to the issue (this is like a regular message, and generates an email too); * when someone posts a PR/review on github, they will also be added automatically on the nosy list of the issue on b.p.o (so they will receive b.p.o messages); * when a PR/review is posted, people in the issue nosy on b.p.o are NOT added to the PR nosy list (they already get new PRs/reviews notifications from b.p.o). This means that: * when a PR is posted: * the PR author will get a mail from b.p.o confirming that the PR has been added to the issue if they have a linked account (possibly on top of any email github might send for creating a PR); * people in the issue nosy on b.p.o will get an email that lets them know a PR has been added; * when a review is posted: * the PR author and the reviewers will get two emails (one from github and one from b.p.o if they have a linked account) or one (from github, if they don't); * people in the issue nosy will only get one email (from b.p.o, with a link to the review); * when a message is posted: * the PR author and reviewers will get one email (from b.p.o if they have a linked account) or zero (if they don't); * people in the issue nosy will get one email (from b.p.o); The "problematic" cases are: * the PR author and the reviewers will receive two emails for reviews if they have a linked account; * the PR author and the reviewers will receive zero emails for b.p.o messages if they don't have a linked account; The former can be solved if we add a way to disable review emails from b.p.o or if github provides one already (or using email filters). The latter can't be solved unless they link accounts (I don't think b.p.o messages should go to github). FWIW I would personally prefer to get all emails from b.p.o, either ignoring/filtering duplicated review emails from github, or disabling them from github if possible. If you agree, this is what needs to be done: 1) automatically add PRs to b.p.o issues; 2) automatically add a message on b.p.o when a review is posted on github; 3) add a "github username" field to b.p.o users to link accounts; 4) automatically add the PR author (during step 1) and reviewers (during step 2) to the issue nosy list on b.p.o; 5) add an option to disable review emails (optional); I would consider all these points except the 5th as blockers. Best Regards, Ezio Melotti From brett at python.org Wed Jan 6 13:18:13 2016 From: brett at python.org (Brett Cannon) Date: Wed, 06 Jan 2016 18:18:13 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Tue, 5 Jan 2016 at 23:46 Ezio Melotti wrote: > On Tue, Jan 5, 2016 at 9:18 PM, Brett Cannon wrote: > > > > > > On Tue, 5 Jan 2016 at 10:48 R. David Murray > wrote: > >> > >> On Tue, 05 Jan 2016 17:50:53 +0000, Brett Cannon > wrote: > >> > If people are that worried, we could do a daily dump of the data. But > >> > unless we can have all GitHub-related comments to an issue not trigger > >> > an > >> > email update I don't think that's feasible as it would mean two emails > >> > for > >> > every PR comment; one from GitHub and one from b.p.o. > >> > >> We can fairly easily have all github-originated comments not trigger an > >> email to the nosy list. We'd just need to add a check for the > >> originating email address to the nosy reaction detector. Even nicer > >> would be to make it a per-user setting (I'd rather get the emails from > >> b.p.o. and disable them on github, myself), but that would be a bit more > >> work. > >> > >> It would be nice to have *all* of the discussion in one place. > > > > > > If someone wants to put the work in to make that happen then that's fine > by > > me, but I do consider this a nicety and not a blocker. We haven't had > code > > review comments go into b.p.o on Rietveld either so this isn't that far > off > > from what we have now, especially if we block the transition on adding a > > link back to the GitHub PR through a web hook. > > I agree it's not a blocker, but a better integration between Rietveld > and b.p.o was often requested (I tried -- and failed -- to tackle the > issue a couple of times). > What I want to avoid is having people on github missing b.p.o > messages, and people on b.p.o missing PRs/reviews on github. (This > was already happening when people were discussing the patch on > Rietveld, and others were only looking at b.p.o and/or missing the > reviews, both because the email from rietveld ended up in the spam > folder, and because there was no way to know from b.p.o if someone > posted a review.) > > I think it's ok to have discussions on b.p.o and reviews on github > (similarly to what we currently do with Rietveld), as long as the > everyone gets notified. Discussions that are not strictly related to > the code/review shouldn't happen on github. > > As for the notifications, I think the best option is: > * when a PR is posted, it's also automatically added to the issue (to > the list of PRs, similar to the list of patches; generates an email > too); > * when a review is posted, a new message with a link is added to the > issue (this is like a regular message, and generates an email too); > * when someone posts a PR/review on github, they will also be added > automatically on the nosy list of the issue on b.p.o (so they will > receive b.p.o messages); > * when a PR/review is posted, people in the issue nosy on b.p.o are > NOT added to the PR nosy list (they already get new PRs/reviews > notifications from b.p.o). > > This means that: > * when a PR is posted: > * the PR author will get a mail from b.p.o confirming that the PR > has been added to the issue if they have a linked account (possibly on > top of any email github might send for creating a PR); > * people in the issue nosy on b.p.o will get an email that lets them > know a PR has been added; > * when a review is posted: > * the PR author and the reviewers will get two emails (one from > github and one from b.p.o if they have a linked account) or one (from > github, if they don't); > * people in the issue nosy will only get one email (from b.p.o, with > a link to the review); > * when a message is posted: > * the PR author and reviewers will get one email (from b.p.o if they > have a linked account) or zero (if they don't); > * people in the issue nosy will get one email (from b.p.o); > > The "problematic" cases are: > * the PR author and the reviewers will receive two emails for reviews > if they have a linked account; > * the PR author and the reviewers will receive zero emails for b.p.o > messages if they don't have a linked account; > > The former can be solved if we add a way to disable review emails from > b.p.o or if github provides one already (or using email filters). The > latter can't be solved unless they link accounts (I don't think b.p.o > messages should go to github). > > FWIW I would personally prefer to get all emails from b.p.o, either > ignoring/filtering duplicated review emails from github, or disabling > them from github if possible. > > If you agree, this is what needs to be done: > 1) automatically add PRs to b.p.o issues; > This is a blocker. > 2) automatically add a message on b.p.o when a review is posted on github; > With the way GitHub does reviews, this won't make sense. Each comment on a PR is essentially its own review (as Eric complained about because that means each comment is treated as a separate thing and thus generates its own email). It isn't like with Rietveld where there is a "Review + Email" step to send draft reviews and hence a review is a group of comments. > 3) add a "github username" field to b.p.o users to link accounts; > That's a blocker for CLA enforcement. > 4) automatically add the PR author (during step 1) and reviewers > (during step 2) to the issue nosy list on b.p.o; > I view this as a great nicety but not a blocker. > 5) add an option to disable review emails (optional); > This will have to be discussed in connection with the emails per PR comment in case there is a misunderstanding of what a review on GitHub is. > > I would consider all these points except the 5th as blockers. > Obviously I don't think they all are, but definitely some. -Brett > > Best Regards, > Ezio Melotti > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Wed Jan 6 14:30:44 2016 From: lac at openend.se (Laura Creighton) Date: Wed, 06 Jan 2016 20:30:44 +0100 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: <568BA75B.6000106@egenix.com> References: <568BA75B.6000106@egenix.com> Message-ID: <201601061930.u06JUiNr025714@theraft.openend.se> In a message of Tue, 05 Jan 2016 12:22:03 +0100, "M.-A. Lemburg" writes: >Given that we want to make it possible to move away from Github >without too much fuzz, wouldn't it be better to have the >PR discussions on b.p.o and Rietvield ? +1 >If we start using Github for this, we'd lose that part of the >review history when moving away. > >Moving from the SF issue tracker to b.p.o was a major piece of work >(mostly done by Martin von L?wis IIRC) and it's not clear how we >could retrofit those discussions into the b.p.o issue discussions. You recall very correctly. Laura From ezio.melotti at gmail.com Thu Jan 7 02:56:20 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 7 Jan 2016 09:56:20 +0200 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Wed, Jan 6, 2016 at 8:18 PM, Brett Cannon wrote: > > > On Tue, 5 Jan 2016 at 23:46 Ezio Melotti wrote: >> ... >> If you agree, this is what needs to be done: >> 1) automatically add PRs to b.p.o issues; > > > This is a blocker. > >> >> 2) automatically add a message on b.p.o when a review is posted on github; > > > With the way GitHub does reviews, this won't make sense. Each comment on a > PR is essentially its own review (as Eric complained about because that > means each comment is treated as a separate thing and thus generates its own > email). It isn't like with Rietveld where there is a "Review + Email" step > to send draft reviews and hence a review is a group of comments. > Good to know. I can think of two alternative solutions: 1) have a "number of comments" and "date/time of the last comment" next to the PR, and update it whenever a new comment is posted; 2) check periodically for new comments, aggregate them somehow, and post a "X new comments have been added to PR Y." message; >> >> 3) add a "github username" field to b.p.o users to link accounts; > > > That's a blocker for CLA enforcement. > >> >> 4) automatically add the PR author (during step 1) and reviewers >> (during step 2) to the issue nosy list on b.p.o; > > > I view this as a great nicety but not a blocker. > OK, I considered this a blocker because otherwise PR authors and reviewers might miss b.p.o comments, but we can let them know manually until it's implemented. Best Regards, Ezio Melotti >> >> 5) add an option to disable review emails (optional); > > > This will have to be discussed in connection with the emails per PR comment > in case there is a misunderstanding of what a review on GitHub is. > >> >> >> I would consider all these points except the 5th as blockers. > > > Obviously I don't think they all are, but definitely some. > > -Brett > >> >> >> Best Regards, >> Ezio Melotti From brett at python.org Thu Jan 7 12:21:37 2016 From: brett at python.org (Brett Cannon) Date: Thu, 07 Jan 2016 17:21:37 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Wed, 6 Jan 2016 at 23:56 Ezio Melotti wrote: > On Wed, Jan 6, 2016 at 8:18 PM, Brett Cannon wrote: > > > > > > On Tue, 5 Jan 2016 at 23:46 Ezio Melotti wrote: > >> ... > >> If you agree, this is what needs to be done: > >> 1) automatically add PRs to b.p.o issues; > > > > > > This is a blocker. > > > >> > >> 2) automatically add a message on b.p.o when a review is posted on > github; > > > > > > With the way GitHub does reviews, this won't make sense. Each comment on > a > > PR is essentially its own review (as Eric complained about because that > > means each comment is treated as a separate thing and thus generates its > own > > email). It isn't like with Rietveld where there is a "Review + Email" > step > > to send draft reviews and hence a review is a group of comments. > > > > Good to know. > I can think of two alternative solutions: > 1) have a "number of comments" and "date/time of the last comment" > next to the PR, and update it whenever a new comment is posted; > 2) check periodically for new comments, aggregate them somehow, and > post a "X new comments have been added to PR Y." message; > Either of those ideas work for me. Would option #2 be done as a comment and thus send out what amounts to a summary email of PR review comments? As long as we do that **only** when there have been changes and not blindly doing it every day (e.g., no "0 comments since yesterday" email), that would be acceptable to me in terms of email volume. Do the other people who wanted to follow PRs on b.p.o like that level of frequency? > > >> > >> 3) add a "github username" field to b.p.o users to link accounts; > > > > > > That's a blocker for CLA enforcement. > > > >> > >> 4) automatically add the PR author (during step 1) and reviewers > >> (during step 2) to the issue nosy list on b.p.o; > > > > > > I view this as a great nicety but not a blocker. > > > > OK, I considered this a blocker because otherwise PR authors and > reviewers might miss b.p.o comments, but we can let them know manually > until it's implemented. > Yep, exactly. I want to block on key stuff that would make GitHub worse than what we have now, but not on stuff that makes it better. This is obviously not to say that I don't want to see all of these great improvements happen, but I do want to get over to the new workflow sooner rather than later as I suspect contributors will find it much easier to work with. -Brett > > Best Regards, > Ezio Melotti > > >> > >> 5) add an option to disable review emails (optional); > > > > > > This will have to be discussed in connection with the emails per PR > comment > > in case there is a misunderstanding of what a review on GitHub is. > > > >> > >> > >> I would consider all these points except the 5th as blockers. > > > > > > Obviously I don't think they all are, but definitely some. > > > > -Brett > > > >> > >> > >> Best Regards, > >> Ezio Melotti > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Thu Jan 7 13:40:19 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 7 Jan 2016 20:40:19 +0200 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Thu, Jan 7, 2016 at 7:21 PM, Brett Cannon wrote: > > > On Wed, 6 Jan 2016 at 23:56 Ezio Melotti wrote: >> >> On Wed, Jan 6, 2016 at 8:18 PM, Brett Cannon wrote: >> > >> > >> > On Tue, 5 Jan 2016 at 23:46 Ezio Melotti wrote: >> >> ... >> >> If you agree, this is what needs to be done: >> >> 1) automatically add PRs to b.p.o issues; >> > >> > >> > This is a blocker. >> > >> >> >> >> 2) automatically add a message on b.p.o when a review is posted on >> >> github; >> > >> > >> > With the way GitHub does reviews, this won't make sense. Each comment on >> > a >> > PR is essentially its own review (as Eric complained about because that >> > means each comment is treated as a separate thing and thus generates its >> > own >> > email). It isn't like with Rietveld where there is a "Review + Email" >> > step >> > to send draft reviews and hence a review is a group of comments. >> > >> >> Good to know. >> I can think of two alternative solutions: >> 1) have a "number of comments" and "date/time of the last comment" >> next to the PR, and update it whenever a new comment is posted; >> 2) check periodically for new comments, aggregate them somehow, and >> post a "X new comments have been added to PR Y." message; > > > Either of those ideas work for me. Would option #2 be done as a comment and > thus send out what amounts to a summary email of PR review comments? As long > as we do that **only** when there have been changes and not blindly doing it > every day (e.g., no "0 comments since yesterday" email), that would be > acceptable to me in terms of email volume. Do the other people who wanted to > follow PRs on b.p.o like that level of frequency? > The goal is to generate at least 1 message/email if a review (possibly comprising several comments) is posted. If there are no new comments, nothing is posted, but if there are new comments, a new message will be posted at some point. We just need to find a compromise between delay and number of messages (and that's something we can figure out later with a bit of trial and error). Checking daily might result in hours of delay, but no more than one daily message. Checking hourly has less delay, but it might result in more messages. We can also try to do something smarter by checking e.g. every 15 minutes and posting the message only if no new messages have been added in the last 15 minutes (so the reviewer has likely finished commenting). >> >> >> >> >> >> 3) add a "github username" field to b.p.o users to link accounts; >> > >> > >> > That's a blocker for CLA enforcement. >> > >> >> >> >> 4) automatically add the PR author (during step 1) and reviewers >> >> (during step 2) to the issue nosy list on b.p.o; >> > >> > >> > I view this as a great nicety but not a blocker. >> > >> >> OK, I considered this a blocker because otherwise PR authors and >> reviewers might miss b.p.o comments, but we can let them know manually >> until it's implemented. > > > Yep, exactly. I want to block on key stuff that would make GitHub worse than > what we have now, but not on stuff that makes it better. This is a good way to see it. Best Regards, Ezio Melotti > This is obviously > not to say that I don't want to see all of these great improvements happen, > but I do want to get over to the new workflow sooner rather than later as I > suspect contributors will find it much easier to work with. > > -Brett > >> >> >> Best Regards, >> Ezio Melotti >> >> >> >> >> 5) add an option to disable review emails (optional); >> > >> > >> > This will have to be discussed in connection with the emails per PR >> > comment >> > in case there is a misunderstanding of what a review on GitHub is. >> > >> >> >> >> >> >> I would consider all these points except the 5th as blockers. >> > >> > >> > Obviously I don't think they all are, but definitely some. >> > >> > -Brett >> > >> >> >> >> >> >> Best Regards, >> >> Ezio Melotti From zachary.ware+pydev at gmail.com Thu Jan 7 14:04:43 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 7 Jan 2016 13:04:43 -0600 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Thu, Jan 7, 2016 at 12:40 PM, Ezio Melotti wrote: > We can also try to do something smarter by checking e.g. every 15 > minutes and posting the message only if no new messages have been > added in the last 15 minutes (so the reviewer has likely finished > commenting). I like this plan, though 15 minutes might be a bit short for the cooldown period; half an hour might be a bit better. On the other hand, it is possible to write out all your review comments without hitting the 'comment' button on any of them, then scroll through and click comment on each of them, in which case the cooldown could be even shorter than 15 minutes. Alternately, what about checking for a marker in a top-level (not a line note) comment, which signals "I finished a review, aggregate it and post it to b.p.o"? -- Zach From brett at python.org Thu Jan 7 14:19:22 2016 From: brett at python.org (Brett Cannon) Date: Thu, 07 Jan 2016 19:19:22 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Thu, 7 Jan 2016 at 11:05 Zachary Ware wrote: > On Thu, Jan 7, 2016 at 12:40 PM, Ezio Melotti > wrote: > > We can also try to do something smarter by checking e.g. every 15 > > minutes and posting the message only if no new messages have been > > added in the last 15 minutes (so the reviewer has likely finished > > commenting). > > I like this plan, though 15 minutes might be a bit short for the > cooldown period; half an hour might be a bit better. On the other > hand, it is possible to write out all your review comments without > hitting the 'comment' button on any of them, then scroll through and > click comment on each of them, in which case the cooldown could be > even shorter than 15 minutes. > > Alternately, what about checking for a marker in a top-level (not a > line note) comment, which signals "I finished a review, aggregate it > and post it to b.p.o"? > The problem is simply forgetting to leave that marker since it isn't required for someone's review comments to be seen in GitHub. -------------- next part -------------- An HTML attachment was scrubbed... URL: From berker.peksag at gmail.com Thu Jan 7 14:27:16 2016 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Thu, 7 Jan 2016 21:27:16 +0200 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Thu, Jan 7, 2016 at 8:40 PM, Ezio Melotti wrote: > The goal is to generate at least 1 message/email if a review (possibly > comprising several comments) is posted. > If there are no new comments, nothing is posted, but if there are new > comments, a new message will be posted at some point. > We just need to find a compromise between delay and number of messages > (and that's something we can figure out later with a bit of trial and > error). > Checking daily might result in hours of delay, but no more than one > daily message. > Checking hourly has less delay, but it might result in more messages. > We can also try to do something smarter by checking e.g. every 15 > minutes and posting the message only if no new messages have been > added in the last 15 minutes (so the reviewer has likely finished > commenting). I think this still will create too much noise. I'd prefer not to see comments like "this needs to be tested", "needs versionadded", "please don't change function signature" etc. in issues. I like following Windows and IDLE issues, but I'm not really interested seeing review comments about them, for example. Wouldn't a new pull request field in the issue detail page be enough to link pull requests? Django uses a similar solution to this: https://code.djangoproject.com/ticket/25995 (see "Pull Requests: 5928 build:success") We could show total number of comments too. From zachary.ware+pydev at gmail.com Thu Jan 7 14:28:55 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 7 Jan 2016 13:28:55 -0600 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: <568BA75B.6000106@egenix.com> <20160105184832.89C59251164@webabinitio.net> Message-ID: On Thu, Jan 7, 2016 at 1:19 PM, Brett Cannon wrote: > On Thu, 7 Jan 2016 at 11:05 Zachary Ware > wrote: >> >> On Thu, Jan 7, 2016 at 12:40 PM, Ezio Melotti >> wrote: >> > We can also try to do something smarter by checking e.g. every 15 >> > minutes and posting the message only if no new messages have been >> > added in the last 15 minutes (so the reviewer has likely finished >> > commenting). >> >> I like this plan, though 15 minutes might be a bit short for the >> cooldown period; half an hour might be a bit better. On the other >> hand, it is possible to write out all your review comments without >> hitting the 'comment' button on any of them, then scroll through and >> click comment on each of them, in which case the cooldown could be >> even shorter than 15 minutes. >> >> Alternately, what about checking for a marker in a top-level (not a >> line note) comment, which signals "I finished a review, aggregate it >> and post it to b.p.o"? > > > The problem is simply forgetting to leave that marker since it isn't > required for someone's review comments to be seen in GitHub. In that case, what about using a marker in combination with a longer cooldown automatic aggregator? If you remember the marker, it posts immediately and is more likely to do what you want; if you forget the marker, it happens eventually anyway. Or just forget the marker entirely, and treat any top-level comment on the PR as "there's something that should be posted to b.p.o now". That method fits better with what we're used to with Rietveld. -- Zach From francismb at email.de Thu Jan 7 15:53:23 2016 From: francismb at email.de (francismb) Date: Thu, 7 Jan 2016 21:53:23 +0100 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: References: Message-ID: <568ED043.1050106@email.de> Hi, On 01/05/2016 07:13 PM, Brett Cannon wrote: > Day 1 summary > ============ > > Decisions made > ----------------------- > > - Seems like our current commit ID -> URL service can be updated to handle > our transition > > > Open issues > ------------------- > - What tools and commands will we use to convert the repos? > - How do we want to handle Misc/NEWS? > - What are the exact commands we are going to expect core devs to run > (which will be documented in the devguide)? > - Who to use for CI (although Donald is the only one to speak up and likes > Travis as do I, so this might be decided)? > - Who to use for code coverage (Donald has suggested codecov)? > - Do we want to add GitHub login support for b.p.o to ease in adding > people's GitHub username and to minimize the need for two accounts to > propose a patch that has a corresponding issue? > a small question: could it be possible to access (some of) the services in a REST-ish form through a wrapper on https://www.python.org/dev I mean something like: Python- community- Service External-Service Python-Website-Address-Wrapper ==================================================================== cpython GitHub https://www.python.org/dev/cpython issues ... https://www.python.org/dev/cpython-issues website GitHub https://www.python.org/dev/website buildbots ... https://www.python.org/dev/cpython-buildbots devinabox GitHub https://www.python.org/dev/cpython-devinabox coverage codecov https://www.python.org/dev/cpython-coverage docs GitHub https://www.python.org/dev/cpython-docs ... Couldn't be then some bots external-provider independent/decoupled? Thanks in advance! francis PS: Sorry if I got some external-service wrong From brett at python.org Thu Jan 7 16:53:21 2016 From: brett at python.org (Brett Cannon) Date: Thu, 07 Jan 2016 21:53:21 +0000 Subject: [core-workflow] My initial thoughts on the steps/blockers of the transition In-Reply-To: <568ED043.1050106@email.de> References: <568ED043.1050106@email.de> Message-ID: On Thu, 7 Jan 2016 at 13:06 francismb wrote: > Hi, > > On 01/05/2016 07:13 PM, Brett Cannon wrote: > > Day 1 summary > > ============ > > > > Decisions made > > ----------------------- > > > > - Seems like our current commit ID -> URL service can be updated to > handle > > our transition > > > > > > Open issues > > ------------------- > > - What tools and commands will we use to convert the repos? > > - How do we want to handle Misc/NEWS? > > - What are the exact commands we are going to expect core devs to run > > (which will be documented in the devguide)? > > - Who to use for CI (although Donald is the only one to speak up and > likes > > Travis as do I, so this might be decided)? > > - Who to use for code coverage (Donald has suggested codecov)? > > - Do we want to add GitHub login support for b.p.o to ease in adding > > people's GitHub username and to minimize the need for two accounts to > > propose a patch that has a corresponding issue? > > > > a small question: could it be possible to access (some of) the services > in a REST-ish form through a wrapper on https://www.python.org/dev > > I mean something like: > > Python- > community- > Service External-Service Python-Website-Address-Wrapper > ==================================================================== > cpython GitHub https://www.python.org/dev/cpython We will probably have git.python.org handle this. > > issues ... https://www.python.org/dev/cpython-issues bugs.python.org already exists. > > website GitHub https://www.python.org/dev/website > buildbots ... https://www.python.org/dev/cpython-buildbots buildbot.python.org already exists. > > devinabox GitHub https://www.python.org/dev/cpython-devinabox It's just a repo and not special enough to deserve a unique URL. > > coverage codecov https://www.python.org/dev/cpython-coverage This might be worth a URL if we get it set up. > > docs GitHub https://www.python.org/dev/cpython-docs That's docs.python.org. > > ... > > > Couldn't be then some bots external-provider independent/decoupled? > The plan is that anything we write for this will be sufficiently decoupled from GitHub that changing hosts will just require updating abstraction code to work with different REST APIs and not a major rewrite of the bot itself. -Brett > > > Thanks in advance! > > francis > > PS: Sorry if I got some external-service wrong > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jan 9 21:54:42 2016 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2016 02:54:42 +0000 Subject: [core-workflow] Migration PEP started Message-ID: I'm developing it at https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . I'm not posting it here as I'm still actively writing it. The only reason I'm mentioning it now is because the migration plan has been very roughly outlined, so if it looks like I'm missing something, please let me know. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Sun Jan 10 03:17:37 2016 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 10 Jan 2016 11:17:37 +0300 Subject: [core-workflow] Migration PEP started In-Reply-To: References: Message-ID: On Sun, Jan 10, 2016 at 5:54 AM, Brett Cannon wrote: > I'm developing it at > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst > . I'm not posting it here as I'm still actively writing it. The only reason > I'm mentioning it now is because the migration plan has been very roughly > outlined, so if it looks like I'm missing something, please let me know. I think that the missing part is the analysis of why community (or PSF, or ...) failed to create a streamlined development process themselves. In particular, what steps were made to implement: 1. online code editing with patch creating through hg.python.org interface 2. HG plugin that can fetch a bugXXX patch and apply it to local copy 3. mercurial queue server to allow people to maintain their own queues of patches in parallel and compare between them 4. what steps were made to make "our fork of the Rietveld code review tool" to be the "our installation" or the "global Roundup service" 5. create testing and building infrastructure (or integrating with CI services such as Drone.IO) for downloading patches, making sure they applied cleanly and running the test suite For simple things like "spelling in documentation" the whole thing with patch production and attachment could be done in 6 months provided that development for 4 people is funded (2 code, 1 frontend, 1 art). The most important stuff - what are the current activities for PSF (or whatever) to being raise funds to make it a paid work instead of relying on a really few core developers and volunteers to do professional work on infra that requires rather tight timely coordination and support? For me, there is a more core problem inside. For example, why work under GSoC 2015 for Roundup was not submitted upstream? Because it was not part of the job. People have problems allocating their free time, because time management practices at their HR departments are getting better and better not leaving much for contributions. I think that the core issue here is the money. The new generation talks about startups and monetization, so if we don't address this, there is little hope that new generation will be attracted, at least that's my observation. I think that if PSF can help us with legal issues concerning funding open source activities, we can construct a few teams through Gratipay and do the development in open way. It will be more effective for Python in the long run, because it will showcase that Python is capable of, and provide people a playground to learn about good engineering practices. I am not saying that this will 100% work - maybe there are things ahead that I do not see as well, but what's drives me mad that it seems that nobody is even trying to preserve the open source spirit in all of that. If we will be sacrificing the open source so easily, then the next Python with static typing will be 'Microsoft Python' or 'Facebook Python' will be rewritten from scratch, and all credits gathered through all these years will be lost behind the shiny name of another corporation. Sad. From victor.stinner at gmail.com Sun Jan 10 04:56:06 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 10 Jan 2016 10:56:06 +0100 Subject: [core-workflow] Migration PEP started In-Reply-To: References: Message-ID: Hi, It looks like there no email to python-dev. Can you please sometimes communicate status on python-dev (and maybe other python mailing list) to keep people aware of what is happening? For example, I don't think that python-dev was involved in the final choice of github. Victor Le dimanche 10 janvier 2016, Brett Cannon a ?crit : > I'm developing it at > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . > I'm not posting it here as I'm still actively writing it. The only reason > I'm mentioning it now is because the migration plan has been very roughly > outlined, so if it looks like I'm missing something, please let me know. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jan 10 12:29:45 2016 From: brett at python.org (Brett Cannon) Date: Sun, 10 Jan 2016 17:29:45 +0000 Subject: [core-workflow] Migration PEP started In-Reply-To: References: Message-ID: Sure, I can drop a note. As for python-dev not being involved, that was somewhat on purpose. I said at the beginning of this process, the switch is geared towards making things easier for core developers while not making things worse for external contributors, and so I focused on python-committers as the most affected group (bringing the whole community in would have just made the process too draining for me). And with Ezio and others planning to put the work in to make sure that non-GitHub contributions are still serviceable I think that goal has been met. On Sun, 10 Jan 2016 at 01:56 Victor Stinner wrote: > Hi, > > It looks like there no email to python-dev. Can you please sometimes > communicate status on python-dev (and maybe other python mailing list) to > keep people aware of what is happening? > > For example, I don't think that python-dev was involved in the final > choice of github. > > Victor > > > Le dimanche 10 janvier 2016, Brett Cannon a ?crit : > >> I'm developing it at >> https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . >> I'm not posting it here as I'm still actively writing it. The only reason >> I'm mentioning it now is because the migration plan has been very roughly >> outlined, so if it looks like I'm missing something, please let me know. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From francismb at email.de Mon Jan 11 15:37:09 2016 From: francismb at email.de (francismb) Date: Mon, 11 Jan 2016 21:37:09 +0100 Subject: [core-workflow] Migration PEP started In-Reply-To: References: Message-ID: <56941275.6090809@email.de> Hi, On 01/10/2016 03:54 AM, Brett Cannon wrote: > I'm developing it at > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . > I'm not posting it here as I'm still actively writing it. The only reason > I'm mentioning it now is because the migration plan has been very roughly > outlined, so if it looks like I'm missing something, please let me know. > my two cents (on changeset 92afd4a90f56a8caa2cf0620cedc2c20b8c0e0d0): """ While hg.python.org [1] hosts many repositories, there are onlyfive key repositories that must move: """ Decide what should happen to the rest of the repos? (or ist under the fait of hg.python.org?) Why is devinabox grey marked, means that something special? """ Define commands to migrate from Mercurial to Git Converting a Mercurial repository to Git """ Aren't both equivalent (or doesn't you need to define the command to be able to convert the repos?). Or you mean write the script for the migration and then do the migration? """ Document steps to commit a pull request Handling Misc/NEWS Handling Misc/ACKS ... """ Couldn't be also part of "Requirements for Code-Only Repositories"? Why only Code-Only repos hasn't Misc/NEWS or Miscs/ACKS? (yes, well those are code only per your definition? :-) ) """ Linking pull requests to issues Notify the issue if the pull request is committed """ There can be more than a pull request to one issue (e.g. an alternative patch), should "the not commited ones" be automaticaly rejected (?) """ Splitting out parts of the documentation into their own repositories """ If those are split, shouldn't exist a mechanism to deploy a consistent release (docs <-> code)? Bot to deploy a release (tagging, building, ...) or at least the steps need to be documented (or will exist a potentially releaseable branch?). Decide and document how the python services map into python.org (e.g. git.python.org) Regards, francis From brett at python.org Mon Jan 11 15:57:35 2016 From: brett at python.org (Brett Cannon) Date: Mon, 11 Jan 2016 20:57:35 +0000 Subject: [core-workflow] Migration PEP started In-Reply-To: <56941275.6090809@email.de> References: <56941275.6090809@email.de> Message-ID: On Mon, 11 Jan 2016 at 12:42 francismb wrote: > Hi, > > On 01/10/2016 03:54 AM, Brett Cannon wrote: > > I'm developing it at > > > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst > . > > I'm not posting it here as I'm still actively writing it. The only reason > > I'm mentioning it now is because the migration plan has been very roughly > > outlined, so if it looks like I'm missing something, please let me know. > > > my two cents (on changeset 92afd4a90f56a8caa2cf0620cedc2c20b8c0e0d0): > > """ > While hg.python.org [1] hosts many repositories, there are onlyfive key > repositories that must move: > """ > Decide what should happen to the rest of the repos? (or ist under > the fait of hg.python.org?) > Part of what happens to hg.python.org since they are all personal repos. > Why is devinabox grey marked, means that something special? > Just that the repo name matches the project name. > > """ > Define commands to migrate from Mercurial to Git > Converting a Mercurial repository to Git > """ > > Aren't both equivalent (or doesn't you need to define the command > to be able to convert the repos?). Or you mean write the script > for the migration and then do the migration? > Accidental duplication. > > """ > Document steps to commit a pull request > Handling Misc/NEWS > Handling Misc/ACKS > ... > """ > Couldn't be also part of "Requirements for Code-Only Repositories"? > Why only Code-Only repos hasn't Misc/NEWS or Miscs/ACKS? (yes, well > those are code only per your definition? :-) ) > Only the cpython repos have an ACKS and NEWS file and all the other repos can just use the merge button in a GitHub PR (that will be pointed out in the cpython section). > > """ > Linking pull requests to issues > Notify the issue if the pull request is committed > """ > There can be more than a pull request to one issue (e.g. an > alternative patch), should "the not committed ones" be automatically > rejected (?) > No, because sometimes an issue involves multiple patches legitimately. > > > """ > Splitting out parts of the documentation into their own repositories > """ > If those are split, shouldn't exist a mechanism to deploy > a consistent release (docs <-> code)? > People are talking about things like the HOWTOs, tutorial, etc. And they can be made to be subrepos of the cpython repo if we want. > > Bot to deploy a release (tagging, building, ...) or at least the > steps need to be documented (or will exist a potentially releaseable > branch?). > There's too much specifically for a bot to do, plus the release process is documented in another PEP. > > > Decide and document how the python services map into python.org > (e.g. git.python.org) > Added a section on creating git.python.org. -Brett > > Regards, > francis > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jan 11 19:25:06 2016 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2016 00:25:06 +0000 Subject: [core-workflow] Migration PEP started In-Reply-To: References: Message-ID: PEP is still not done, but I have at least fleshed out the parts required to migrate the code-only and web-related repositories: https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . If you find typos and such in the PEP, feel free to send a pull request (thanks to Zachary for already doing this!). Once the PEP no longer has XXX markers I will post the text to this mailing list (aiming for this week, or at worst sometime next weekend). On Sat, 9 Jan 2016 at 18:54 Brett Cannon wrote: > I'm developing it at > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . > I'm not posting it here as I'm still actively writing it. The only reason > I'm mentioning it now is because the migration plan has been very roughly > outlined, so if it looks like I'm missing something, please let me know. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue Jan 12 02:35:35 2016 From: phd at phdru.name (Oleg Broytman) Date: Tue, 12 Jan 2016 08:35:35 +0100 Subject: [core-workflow] Migration PEP started In-Reply-To: References: Message-ID: <20160112073535.GA21828@phdru.name> Hi! On Tue, Jan 12, 2016 at 12:25:06AM +0000, Brett Cannon wrote: > PEP is still not done, but I have at least fleshed out the parts required > to migrate the code-only and web-related repositories: > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . > Continuous integration per pull request > XXX https://travis-ci.org/ https://codeship.com/ https://circleci.com/ No buildbot? > Tools and commands to move from Mercurial to Git > A decision needs to be made on exactly what tooling and what commands > involving those tools will be used to convert a Mercurial repository to > Git. Currently a suggestion has been made to use > https://github.com/frej/fast-export. My personal experience with fast-export was a bit negative when I tried to convert a repository for a complex project with a number of branches and a lot of tags. I switched to git-remote-hg [1] and found it much better for my needs. See Comparison [2]. 1. https://github.com/felipec/git-remote-hg 2. https://github.com/felipec/git/wiki/Comparison-of-git-remote-hg-alternatives Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From brett at python.org Tue Jan 12 11:32:05 2016 From: brett at python.org (Brett Cannon) Date: Tue, 12 Jan 2016 16:32:05 +0000 Subject: [core-workflow] Migration PEP started In-Reply-To: <20160112073535.GA21828@phdru.name> References: <20160112073535.GA21828@phdru.name> Message-ID: On Mon, 11 Jan 2016 at 23:52 Oleg Broytman wrote: > Hi! > > On Tue, Jan 12, 2016 at 12:25:06AM +0000, Brett Cannon > wrote: > > PEP is still not done, but I have at least fleshed out the parts required > > to migrate the code-only and web-related repositories: > > > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst > . > > > Continuous integration per pull request > > XXX https://travis-ci.org/ https://codeship.com/ https://circleci.com/ > > No buildbot? > Buildbot serves a different purpose. This unfinished item is for CI for every submitted pull request while Buildbot is for verifying that the repository on key branches is in a good state. > > > Tools and commands to move from Mercurial to Git > > A decision needs to be made on exactly what tooling and what commands > > involving those tools will be used to convert a Mercurial repository to > > Git. Currently a suggestion has been made to use > > https://github.com/frej/fast-export. > > My personal experience with fast-export was a bit negative when I > tried to convert a repository for a complex project with a number of > branches and a lot of tags. I switched to git-remote-hg [1] and found it > much better for my needs. See Comparison [2]. > > 1. https://github.com/felipec/git-remote-hg > 2. > https://github.com/felipec/git/wiki/Comparison-of-git-remote-hg-alternatives I'll add it to the list of options. -Brett > > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From francismb at email.de Sat Jan 16 11:33:19 2016 From: francismb at email.de (francismb) Date: Sat, 16 Jan 2016 17:33:19 +0100 Subject: [core-workflow] Migration PEP started In-Reply-To: References: Message-ID: <569A70CF.6090004@email.de> Hi Brett, On 01/10/2016 03:54 AM, Brett Cannon wrote: > I'm developing it at > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst . > I'm not posting it here as I'm still actively writing it. The only reason > I'm mentioning it now is because the migration plan has been very roughly > outlined, so if it looks like I'm missing something, please let me know. > > A question on point: Linking pull requests to issues =============================== Historically external contributions were attached to an issue on bugs.python.org [5] thanks to the fact that all external contributions were uploaded as a file. [...] * From GitHub [1]: Deleting your user account You can delete your GitHub user account at any time. Before you do so, you should hand over the reins of any organizations you might own. Deleting your user account removes all repositories, forks of private repositories, wikis, issues, pull requests, and pages. The account name also becomes available to anyone else to use on a new account, and we stop billing you. Is that case solved under the point """Backup of pull request data"""? (or where should point the issue-tracker if the user is gone?) Thanks, francis ---- [1] https://help.github.com/articles/deleting-your-user-account/ From brett at python.org Sat Jan 16 13:08:36 2016 From: brett at python.org (Brett Cannon) Date: Sat, 16 Jan 2016 18:08:36 +0000 Subject: [core-workflow] Migration PEP started In-Reply-To: <569A70CF.6090004@email.de> References: <569A70CF.6090004@email.de> Message-ID: On Sat, 16 Jan 2016 at 08:33 francismb wrote: > Hi Brett, > > On 01/10/2016 03:54 AM, Brett Cannon wrote: > > I'm developing it at > > > https://github.com/brettcannon/github-transition-pep/blob/master/pep-NNNN.rst > . > > I'm not posting it here as I'm still actively writing it. The only reason > > I'm mentioning it now is because the migration plan has been very roughly > > outlined, so if it looks like I'm missing something, please let me know. > > > > > > A question on point: > > Linking pull requests to issues > =============================== > Historically external contributions were attached to an issue on > bugs.python.org [5] thanks to the fact that all external contributions > were uploaded as a file. > [...] > > > * From GitHub [1]: > > Deleting your user account > > You can delete your GitHub user account at any time. Before you do so, > you should hand over the reins of any organizations you might own. > > Deleting your user account removes all repositories, forks of private > repositories, wikis, issues, pull requests, and pages. The account name > also becomes available to anyone else to use on a new account, and we > stop billing you. > > > Is that case solved under the point """Backup of pull request data"""? > (or where should point the issue-tracker if the user is gone?) > I'm not terribly worried about this case. It's going to be rare and if someone walks away from their contribution they may not want us to carry on. It should be said, though, that some have talked about backing up the diffs so it's definitely feasible to save all of it and not just the comments. -Brett > > > Thanks, > > francis > > ---- > [1] https://help.github.com/articles/deleting-your-user-account/ > > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jan 17 15:48:42 2016 From: brett at python.org (Brett Cannon) Date: Sun, 17 Jan 2016 20:48:42 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub Message-ID: Here is the first complete draft of the PEP laying out what needs to happen to migrate to GitHub (along with some extras that would be nice). The PEP can also be viewed at https://github.com/brettcannon/github-transition-pep/blob/master/pep-0512.rst or at https://www.python.org/dev/peps/ once the PEP index is updated. My hope is that I captured all of the requirements for the various repositories. If we can agree that is the case then we can start work on the various requirements and start the migration work! ---------- PEP: 512 Title: Migrating from hg.python.org to GitHub Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Status: Active Type: Process Content-Type: text/x-rst Created: Post-History: 17-Jan-2015 Abstract ======== This PEP outlines the steps required to migrate Python's development process from Mercurial [#hg]_ as hosted at hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting the minimum goals of this PEP should allow for the development process of Python to be as productive as it currently is, and meeting its extended goals should improve it. Rationale ========= In 2014, it became obvious that Python's custom development process was becoming a hindrance. As an example, for an external contributor to submit a fix for a bug that eventually was committed, the basic steps were: 1. Open an issue for the bug at bugs.python.org [#b.p.o]_. 2. Checkout out the CPython source code from hg.python.org [#h.p.o]_. 3. Make the fix. 4. Upload a patch. 5. Have a core developer review the patch using our fork of the Rietveld code review tool [#rietveld]_. 6. Download the patch to make sure it still applies cleanly. 7. Run the test suite manually. 8. Commit the change manually. 9. If the change was for a bugfix release, merge into the in-development branch. 10. Run the test suite manually again. 11. Commit the merge. 12. Push the changes. This is a very heavy, manual process for core developers. Even in the simple case, you could only possibly skip the code review step, as you would still need to build the documentation. This led to patches languishing on the issue tracker due to core developers not being able to work through the backlog fast enough to keep up with submissions. In turn, that led to a side-effect issue of discouraging outside contribution due to frustration from lack of attention, which is dangerous problem for an open source project as it runs counter to having a viable future for the project. Simply accepting patches uploaded to bugs.python.org [#b.p.o]_ is potentially simple for an external contributor, it is as slow and burdensome as it gets for a core developer to work with. Hence the decision was made in late 2014 that a move to a new development process was needed. A request for PEPs proposing new workflows was made, in the end leading to two: PEP 481 and PEP 507 proposing GitHub [#github]_ and GitLab [#gitlab]_, respectively. The year 2015 was spent off-and-on working on those proposals and trying to tease out details of what made them different from each other on the core-workflow mailing list [#core-workflow]_. PyCon US 2015 also showed that the community was a bit frustrated with our process due to both cognitive overhead for new contributors and how long it was taking for core developers to look at a patch (see the end of Guido van Rossum's keynote at PyCon US 2015 [#guido-keynote]_ as an example of the frustration). On January 1, 2016, the decision was made by Brett Cannon to move the development process to GitHub. The key reasons for choosing GitHub were [#reasons]_: * Maintaining custom infrastructure has been a burden on volunteers (e.g., a custom fork of Rietveld [#rietveld]_ that is not being maintained is currently being used). * The custom workflow is very time-consuming for core developers (not enough automated tooling built to help support it). * The custom workflow is a hindrance to external contributors (acts as a barrier of entry due to time required to ramp up on development process). * There is no feature differentiating GitLab from GitHub beyond GitLab being open source. * Familiarity with GitHub is far higher amongst core developers and external contributors than with GitLab. * Our BDFL prefers GitHub (who would be the first person to tell you that his opinion shouldn't matter, but the person making the decision felt it was important that the BDFL feel comfortable with the workflow of his own programming language to encourage his continued participation). There's even already an unofficial image to use to represent the migration to GitHub [#pythocat]_. The overarching goal of this migration is to improve the development process to the extent that a core developer can go from external contribution submission through all the steps leading to committing said contribution all from within a browser on a tablet with WiFi. All of this will be done in such a way that if an external contributor chooses not to use GitHub then they will continue to have that option. Repositories to Migrate ======================= While hg.python.org [#h.p.o]_ hosts many repositories, there are only six key repositories that must move: 1. devinabox [#devinabox-repo]_ 2. benchmarks [#benchmarks-repo]_ 3. tracker [#tracker-repo]_ 4. peps [#peps-repo]_ 5. devguide [#devguide-repo]_ 6. cpython [#cpython-repo]_ The devinabox, benchmarksm and tracker repositories are code-only. The peps and devguide repositories involve the generation of webpages. And the cpython repository has special requirements for integration with bugs.python.org [#b.p.o]_. Migration Plan ============== The migration plan is separated into sections based on what is required to migrate the repositories listed in the `Repositories to Migrate`_ section. Completion of requirements outlined in each section should unblock the migration of the related repositories. The sections are expected to be completed in order, but not necessarily the requirements within a section. Requirements for Code-Only Repositories --------------------------------------- Completion of the requirements in this section will allow the devinabox, benchmarks, and tracker repositories to move to GitHub. While devinabox has a sufficiently descriptive name, the benchmarks repository does not; therefore, it will be named "python-benchmark-suite". The tracker repo also has a misleading name and will be renamed "bugs.python.org". Create a 'python-dev' team '''''''''''''''''''''''''' To manage permissions, a 'python-dev' team will be created as part of the python organization [#github-python-org]_. Any repository that is moved will have the 'python-dev' team added to it with write permissions [#github-org-perms]_. Anyone who previously had rights to manage SSH keys on hg.python.org will become a team maintainer for the 'python-dev' team. Define commands to move a Mercurial repository to Git ''''''''''''''''''''''''''''''''''''''''''''''''''''' Since moving to GitHub also entails moving to Git [#git]_, we must decide what tools and commands we will run to translate a Mercurial repository to Git. The exact tools and steps to use are an open issue; see `Tools and commands to move from Mercurial to Git`_. CLA enforcement ''''''''''''''' A key part of any open source project is making sure that its source code can be properly licensed. This requires making sure all people making contributions have signed a contributor license agreement (CLA) [#cla]_. Up until now, enforcement of CLA signing of contributed code has been enforced by core developers checking whether someone had an ``*`` by their username on bugs.python.org [#b.p.o]_. With this migration, the plan is to start off with automated checking and enforcement of contributors signing the CLA. Adding GitHub username support to bugs.python.org +++++++++++++++++++++++++++++++++++++++++++++++++ To keep tracking of CLA signing under the direct control of the PSF, tracking who has signed the PSF CLA will be continued by marking that fact as part of someone's bugs.python.org user profile. What this means is that an association will be needed between a person's bugs.python.org [#b.p.o]_ account and their GitHub account, which will be done through a new field in a user's profile. A bot to enforce CLA signing ++++++++++++++++++++++++++++ With an association between someone's GitHub account and their bugs.python.org [#b.p.o]_ account, which has the data as to whether someone has signed the CLA, a bot can monitor pull requests on GitHub and denote whether the contributor has signed the CLA. If the user has signed the CLA, the bot will add a positive label to the issue to denote the pull request has no CLA issues (e.g., a green label stating, "CLA: ?"). If the contributor has not signed a CLA, a negative label will be added to the pull request will be blocked using GitHub's status API (e.g., a red label stating, "CLA: ?"). If a contributor lacks a bugs.python.org account, that will lead to another label (e.g., "CLA: ? (no account)"). Using a label for both positive and negative cases provides a fallback notification if the bot happens to fail, preventing potential false-positives or false-negatives. It also allows for an easy way to trigger the bot again by simply removing a CLA-related label. Requirements for Web-Related Repositories ----------------------------------------- Due to their use for generating webpages, the devguide [#devguide-repo]_ and peps [#peps-repo]_ repositories need their respective processes updated to pull from their new Git repositories. The devguide repository might also need to be named ``python-devguide`` to make sure the repository is not ambiguous when viewed in isolation from the python organization [#github-python-org]_. Requirements for the cpython Repository --------------------------------------- Obviously the most active and important repository currently hosted at hg.python.org [#h.p.o]_ is the cpython repository [#cpython-repo]_. Because of its importance and high- frequency use, it requires more tooling before being moved to GitHub compared to the other repositories mentioned in this PEP. Document steps to commit a pull request ''''''''''''''''''''''''''''''''''''''' During the process of choosing a new development workflow, it was decided that a linear history is desired. This means that the convenient "Merge" button in GitHub pull requests is undesireable, as it creates a merge commit along with all of the contributor's individual commits (this does not affect the other repositories where the desire for a linear history doesn't exist). Luckily, Git [#git]_ does not require GitHub's workflow and so one can be chosen which gives us a linear history by using Git's CLI. The expectation is that all pull requests will be fast-forwarded and rebased before being pushed to the master repository. This should give proper attribution to the pull request author in the Git history. A second set of recommended commands will also be written for committing a contribution from a patch file uploaded to bugs.python.org [#b.p.o]_. This will obviously help keep the linear history, but it will need to be made to have attribution to the patch author. The exact sequence of commands that will be given as guidelines to core developers is an open issue: `Git CLI commands for committing a pull request to cpython`_. Handling Misc/NEWS '''''''''''''''''' Traditionally the ``Misc/NEWS`` file [#news-file]_ has been problematic for changes which spanned Python releases. Often times there will be merge conflicts when committing a change between e.g., 3.5 and 3.6 only in the ``Misc/NEWS`` file. It's so common, in fact, that the example instructions in the devguide explicitly mention how to resolve conflicts in the ``Misc/NEWS`` file [#devguide-merge-across-branches]_. As part of our tool modernization, working with the ``Misc/NEWS`` file will be simplified. There are currently two competing approaches to solving the ``Misc/NEWS`` problem which are discussed in an open issue: `How to handle the Misc/NEWS file`_. Handling Misc/ACKS '''''''''''''''''' Traditionally the ``Misc/ACKS`` file [#acks-file]_ has been managed by hand. But thanks to Git supporting an ``author`` value as well as a ``committer`` value per commit, authorship of a commit can be part of the history of the code itself. As such, manual management of ``Misc/ACKS`` will become optional. A script will be written that will collect all author and committer names and merge them into ``Misc/ACKS`` with all of the names listed prior to the move to Git. Running this script will become part of the release process. Linking pull requests to issues ''''''''''''''''''''''''''''''' Historically, external contributions were attached to an issue on bugs.python.org [#b.p.o]_ thanks to the fact that all external contributions were uploaded as a file. For changes committed by a core developer who committed a change directly, the specifying of an issue number in the commit message of the format ``Issue #`` at the start of the message led to a comment being posted to the issue linking to the commit. Linking a pull request to an issue ++++++++++++++++++++++++++++++++++ An association between a pull request and an issue is needed to track when a fix has been proposed. The association needs to be many-to-one as there can take multiple pull requests to solve a single issue (technically it should be a many-to-many association for when a single fix solves multiple issues, but this is fairly rare and issues can be merged into one using the ``Superceder`` field on the issue tracker). Association between a pull request and an issue will be done based on detecting the regular expression``[Ii]ssue #(?P\d+)``. If this is specified in either the title or in the body of a message on a pull request then connection will be made on bugs.python.org [#b.p.o]_. A label will also be added to the pull request to signify that the connection was made successfully. This could lead to incorrect associations if the wrong issue or referencing another issue was done, but these are rare occasions. Notify the issue if the pull request is committed +++++++++++++++++++++++++++++++++++++++++++++++++ Once a pull request is closed (merged or not), the issue should be updated to reflect this fact. Update linking service for mapping commit IDs to URLs ''''''''''''''''''''''''''''''''''''''''''''''''''''' Currently you can use https://hg.python.org/lookup/ with a revision ID from either the Subversion or Mercurial copies of the cpython repo [#cpython-repo]_ to get redirected to the URL for that revision in the Mercurial repository. The URL rewriter will need to be updated to redirect to the Git repository and to support the new revision IDs created for the Git repository. Create https://git.python.org ''''''''''''''''''''''''''''' Just as hg.python.org [#h.p.o]_ currently points to the Mercurial repository for Python, git.python.org should do the equivalent for the Git repository. Backup of pull request data ''''''''''''''''''''''''''' Since GitHub [#github]_ is going to be used for code hosting and code review, those two things need to be backed up. In the case of code hosting, the backup is implicit as all non-shallow Git [#git]_ clones contain the full history of the repository, hence there will be many backups of the repository. The code review history does not have the same implicit backup mechanism as the repository itself. That means a daily backup of code review history should be done so that it is not lost in case of any issues with GitHub. It also helps guarantee that a migration from GitHub to some other code review system is feasible were GitHub to disappear overnight. Change sys._mercurial ''''''''''''''''''''' Once Python is no longer kept in Mercurial, the ``sys._mercurial`` attribute will need to be removed. An equivalent ``sys._git`` attribute will be needed to take its place. Optional, Planned Features -------------------------- Once the cpython repository [#cpython-repo]_ is migrated, all repositories will have been moved to GitHub [#github]_ and the development process should be on equal footing as before. But a key reason for this migration is to improve the development process, making it better than it has ever been. This section outlines some plans on how to improve things. It should be mentioned that overall feature planning for bugs.python.org [#b.p.o]_ -- which includes plans independent of this migration -- are tracked on their own wiki page [#tracker-plans]_. Bot to handle pull request merging '''''''''''''''''''''''''''''''''' As stated in the section entitled "`Document steps to commit a pull request`_", the desire is to maintain a linear history for cpython. Unfortunately, Github's [#github]_ web-based workflow does not support a linear history. Because of this, a bot should be written to substitute for GitHub's in-browser commit abilities. To start, the bot should accept commands to commit a pull request against a list of branches. This allows for committing a pull request that fixes a bug in multiple versions of Python. More advanced features such as a commit queue can come later. This would linearly apply accepted pull requests and verify that the commits did not interfere with each other by running the test suite and backing out commits if the test run failed. To help facilitate the speed of testing, all patches committed since the last test run can be applied and run in a single test run as the optimistic assumption is that the patches will work in tandem. Inspiration or basis of the bot could be taken from pre-existig bots such as Homu [#homu]_ or Zuul [#zuul]_. The name given to this bot in order to give it commands is an open issue: `Naming the commit bot`_. Continuous integration per pull request ''''''''''''''''''''''''''''''''''''''' To help speed up pull request approvals, continuous integration testing should be used. This helps mitigate the need for a core developer to download a patch simply to run the test suite against the patch. Which free CI service to use is an open issue: `Choosing a CI service`_. Test coverage report '''''''''''''''''''' Getting an up-to-date test coverage report for Python's standard library would be extremely beneficial as generating such a report can take quite a while to produce. There are a couple pre-existing services that provide free test coverage for open source projects. Which option is best is an open issue: `Choosing a test coverage service`_. Notifying issues of pull request comments ''''''''''''''''''''''''''''''''''''''''' The current development process does not include notifying an issue on bugs.python.org [#b.p.o]_ when a review comment is left on Rietveld [#rietveld]_. It would be nice to fix this so that people can subscribe only to comments at bugs.python.org and not GitHub [#github]_ and yet still know when something occurs on GitHub in terms of review comments on relevant pull requests. Current thinking is to post a comment to bugs.python.org to the relevant issue when at least one review comment has been made over a certain period of time (e.g., 15 or 30 minutes). This keeps the email volume down for those that receive both GitHub and bugs.python.org email notifications while still making sure that those only following bugs.python.org know when there might be a review comment to address. Allow bugs.python.org to use GitHub as a login provider ''''''''''''''''''''''''''''''''''''''''''''''''''''''' As of right now, bugs.python.org [#b.p.o]_ allows people to log in using Google, Launchpad, or OpenID credentials. It would be good to expand this to GitHub credentials. Web hooks for re-generating web content ''''''''''''''''''''''''''''''''''''''' The content at https://docs.python.org/, https://docs.python.org/devguide, and https://www.python.org/dev/peps/ are all derived from files kept in one of the repositories to be moved as part of this migration. As such, it would be nice to set up appropriate webhooks to trigger rebuilding the appropriate web content when the files they are based on change instead of having to wait for, e.g., a cronjob to trigger. Link web content back to files that it is generated from '''''''''''''''''''''''''''''''''''''''''''''''''''''''' It would be helpful for people who find issues with any of the documentation that is generated from a file to have a link on each page which points back to the file on GitHub [#github]_ that stores the content of the page. That would allow for quick pull requests to fix simple things such as spelling mistakes. Splitting out parts of the documentation into their own repositories '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' While certain parts of the documentation at https://docs.python.org change with the code, other parts are fairly static and are not tightly bound to the CPython code itself. The following sections of the documentation fit this category of slow-changing, loosely-coupled: * `Tutorial `__ * `Python Setup and Usage `__ * `HOWTOs `__ * `Installing Python Modules < https://docs.python.org/3/installing/index.html>`__ * `Distributing Python Modules < https://docs.python.org/3/distributing/index.html>`__ * `Extending and Embedding `__ * `FAQs `__ These parts of the documentation could be broken out into their own repositories to simplify their maintenance and to expand who has commit rights to them to ease in their maintenance. Status ====== Requirements for migrating the devinabox [#devinabox-repo]_, benchmarks [#benchmarks-repo]_, and tracker [#tracker-repo]_ repositories: * Not started - `Create a 'python-dev' team`_ - `Define commands to move a Mercurial repository to Git`_ - `Adding GitHub username support to bugs.python.org`_ - `A bot to enforce CLA signing`_ * In progress - None * Completed - None Repositories whose build steps need updating: * Not started - peps [#peps-repo]_ - devguide [#devguide-repo]_ * In progress - None * Completed - None Requirements to move over the cpython repo [#cpython-repo]_: * Not started - `Document steps to commit a pull request`_ - `Handling Misc/NEWS`_ - `Handling Misc/ACKS`_ - `Linking a pull request to an issue`_ - `Notify the issue if the pull request is committed`_ - `Update linking service for mapping commit IDs to URLs`_ - `Create https://git.python.org`_ - `Backup of pull request data`_ - `Change sys._mercurial`_ * In progress - None * Completed - None Optional features: * Not started - `Bot to handle pull request merging`_ - `Continuous integration per pull request`_ - `Test coverage report`_ - `Notifying issues of pull request comments`_ - `Allow bugs.python.org to use GitHub as a login provider`_ - `Web hooks for re-generating web content`_ - `Link web content back to files that it is generated from`_ - `Splitting out parts of the documentation into their own repositories`_ * In progress - None * Completed - None Open Issues =========== For this PEP, open issues are ones where a decision needs to be made to how to approach or solve a problem. Open issues do not entail coordination issues such as who is going to write a certain bit of code. The fate of hg.python.org ------------------------- With the code repositories moving over to Git [#git]_, there is no technical need to keep hg.python.org [#h.p.o]_ running. Having said that, some in the community would like to have it stay functioning as a Mercurial [#hg]_ mirror of the Git repositories. Others have said that they still want a mirror, but one using Git. As maintaining hg.python.org is not necessary, it will be up to the PSF infrastructure committee to decide if they want to spend the time and resources to keep it running. They may also choose whether they want to host a Git mirror on PSF infrastructure. Tools and commands to move from Mercurial to Git ------------------------------------------------ A decision needs to be made on exactly what tooling and what commands involving those tools will be used to convert a Mercurial repository to Git. Currently a suggestion has been made to use https://github.com/frej/fast-export. Another suggestion is to use https://github.com/felipec/git-remote-hg. Finally, http://hg-git.github.io/ has been suggested. Git CLI commands for committing a pull request to cpython --------------------------------------------------------- Because Git [#git]_ may be a new version control system for core developers, the commands people are expected to run will need to be written down. These commands also need to keep a linear history while giving proper attribution to the pull request author. Another set of commands will also be necessary for when working with a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear history will be kept implicitly, but it will need to make sure to keep/add attribution. How to handle the Misc/NEWS file -------------------------------- There are two competing approaches to handling ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on bugs.python.org [#b.p.o]_. This would mean an issue that is marked as "resolved" could not be closed until a news entry is added in the "news" field in the issue tracker. The benefit of tying the news entry to the issue is it makes sure that all changes worthy of a news entry have an accompanying issue. It also makes classifying a news entry automatic thanks to the Component field of the issue. The Versions field of the issue also ties the news entry to which Python releases were affected. A script would be written to query bugs.python.org for relevant new entries for a release and to produce the output needed to be checked into the code repository. This approach is agnostic to whether a commit was done by CLI or bot. The competing approach is to use an individual file per news entry, containg the text for the entry. In this scenario each feature release would have its own directory for news entries and a separate file would be created in that directory that was either named after the issue it closed or a timestamp value (which prevents collisions). Merges across branches would have no issue as the news entry file would still be uniqeuely named and in the directory of the latest version that contained the fix. A script would collect all news entry files no matter what directory they reside in and create an appropriate news file (the release directory can be ignored as the mere fact that the file exists is enough to represent that the entry belongs to the release). Classification can either be done by keyword in the new entry file itself or by using subdirectories representing each news entry classification in each release directory (or classification of news entries could be dropped since critical information is captured by the "What's New" documents which are organized). The benefit of this approach is that it keeps the changes with the code that was actually changed. It also ties the message to being part of the commit which introduced the change. For a commit made through the CLI, a script will be provided to help generate the file. In a bot-driven scenario, the merge bot will have a way to specify a specific news entry and create the file as part of its flattened commit (while most likely also supporting using the first line of the commit message if no specific news entry was specified). Code for this approach has been written previously for the Mercurial workflow at http://bugs.python.org/issue18967. There is also a tool from the community at https://pypi.python.org/pypi/towncrier. Naming the commit bot --------------------- As naming things can lead to bikeshedding of epic proportions, Brett Cannon will choose the final name of the commit bot (the name of the project for the bot itself can be anything, this is purely for the name used in giving commands to the bot). The name will come from Monty Python, which is only fitting since Python is named after the comedy troupe. It will most likely come from 'Monty Python and the Holy Grail' [#holy-grail]_ (which happens to be how Brett was introduced to Monty Python). Current ideas on the name include: "Black Knight" sketch [#black-knight-sketch]_: * black-knight * none-shall-pass * just-a-flesh-wound "Bridge of Death" sketch [#bridge-of-death-sketch]_: * bridge-keeper * man-from-scene-24 * five-questions * what-is-your-quest * blue-no-green * air-speed-velocity * your-favourite-colour (and that specific spelling; Monty Python is British, after all) "Killer rabbit" sketch [#killer-rabbit-sketch]_: * killer-rabbit * holy-hand-grenade * 5-is-right-out "Witch Village" sketch [#witch-village-sketch]_: * made-of-wood * burn-her "French Taunter" sketch [#french-taunter-sketch]_: * elderberries * kanigget "Constitutional Peasants" sketch [#constitutional-peasants-sketch]_: * dennis * from-the-masses * watery-tart "Knights Who Say Ni" sketch [#ni-sketch]_: * shubbery * ni >From "Monty Python and the Holy Grail" in general: * brave-sir-robin Choosing a CI service --------------------- There are various CI services that provide free support for open source projects hosted on GitHub [#github]_. Two such examples are Travis [#travis]_ and Codeship [#codeship]_. Whatever solution is chosen will need to not time out in the time it takes to execute Python's test suite. It should optimally provide access to multiple C compilers for more thorough testing. Network access is also beneficial. The current CI service for Python is Pypatcher [#pypatcher]_. A request can be made in IRC to try a patch from bugs.python.org [#b.p.o]_. The results can be viewed at https://ci.centos.org/job/cPython-build-patch/ . Choosing a test coverage service -------------------------------- Getting basic test coverage of Python's standard library can be created simply by using coverage.py [#coverage]_. Getting thorough test coverage is actually quite tricky, with the details outlined in the devinabox's README [#devinabox-repo]_. It would be best if a service could be found that would allow for thorough test coverage, but it might not be feasible. Free test coverage services include Coveralls [#coveralls]_ and Codecov [#codecov]_. Rejected Ideas ============== Separate Python 2 and Python 3 repositories ------------------------------------------- It was discussed whether separate repositories for Python 2 and Python 3 were desired. The thinking was that this would shrink the overall repository size which benefits people with slow Internet connections or small bandwidth caps. In the end it was decided that it was easier logistically to simply keep all of CPython's history in a single repository. Commit multi-release changes in bugfix branch first --------------------------------------------------- As the current development process has changes committed in the oldest branch first and then merged up to the default branch, the question came up as to whether this workflow should be perpetuated. In the end it was decided that committing in the newest branch and then cherrypicking changes into older branches would work best as most people will instinctively work off the newest branch and it is a more common workflow when using Git [#git]_. Deriving ``Misc/NEWS`` from the commit logs ------------------------------------------- As part of the discussion surrounding `Handling Misc/NEWS`_, the suggestion has come up of deriving the file from the commit logs itself. In this scenario, the first line of a commit message would be taken to represent the news entry for the change. Some heuristic to tie in whether a change warranted a news entry would be used, e.g., whether an issue number is listed. This idea has been rejected due to some core developers preferring to write a news entry separate from the commit message. The argument is the first line of a commit message compared to that of a news entry have different requirements in terms of brevity, what should be said, etc. References ========== .. [#h.p.o] https://hg.python.org .. [#GitHub] GitHub (https://github.com) .. [#hg] Mercurial (https://www.mercurial-scm.org/) .. [#git] Git (https://git-scm.com/) .. [#b.p.o] https://bugs.python.org .. [#rietveld] Rietveld (https://github.com/rietveld-codereview/rietveld) .. [#gitlab] GitLab (https://about.gitlab.com/) .. [#core-workflow] core-workflow mailing list ( https://mail.python.org/mailman/listinfo/core-workflow) .. [#guido-keynote] Guido van Rossum's keynote at PyCon US ( https://www.youtube.com/watch?v=G-uKNd5TSBw) .. [#reasons] Email to core-workflow outlining reasons why GitHub was selected (https://mail.python.org/pipermail/core-workflow/2016-January/000345.html ) .. [#benchmarks-repo] Mercurial repository for the Unified Benchmark Suite (https://hg.python.org/benchmarks/) .. [#devinabox-repo] Mercurial repository for devinabox ( https://hg.python.org/devinabox/) .. [#peps-repo] Mercurial repository of the Python Enhancement Proposals ( https://hg.python.org/peps/) .. [#tracker-repo] bugs.python.org code repository ( https://hg.python.org/tracker/python-dev/) .. [#devguide-repo] Mercurial repository for the Python Developer's Guide ( https://hg.python.org/devguide/) .. [#cpython-repo] Mercurial repository for CPython ( https://hg.python.org/cpython/) .. [#github-python-org] Python organization on GitHub ( https://github.com/python) .. [#github-org-perms] GitHub repository permission levels ( https://help.github.com/enterprise/2.4/user/articles/repository-permission-levels-for-an-organization/ ) .. [#cla] Python Software Foundation Contributor Agreement ( https://www.python.org/psf/contrib/contrib-form/) .. [#news-file] ``Misc/NEWS`` ( https://hg.python.org/cpython/file/default/Misc/NEWS) .. [#acks-file] ``Misc/ACKS`` ( https://hg.python.org/cpython/file/default/Misc/ACKS) .. [#devguide-merge-across-branches] Devguide instructions on how to merge across branches ( https://docs.python.org/devguide/committing.html#merging-between-different-branches-within-the-same-major-version ) .. [#pythocat] Pythocat (https://octodex.github.com/pythocat/) .. [#tracker-plans] Wiki page for bugs.python.org feature development (https://wiki.python.org/moin/TrackerDevelopmentPlanning) .. [#black-knight-sketch] The "Black Knight" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=dhRUe-gz690) .. [#bridge-of-death-sketch] The "Bridge of Death" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=cV0tCphFMr8) .. [#holy-grail] "Monty Python and the Holy Grail" sketches (https://www.youtube.com/playlist?list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp ) .. [#killer-rabbit-sketch] "Killer rabbit" sketch from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=Nvs5pqf-DMA&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=11 ) .. [#witch-village-sketch] "Witch Village" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=k3jt5ibfRzw&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=12 ) .. [#french-taunter-sketch] "French Taunter" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=A8yjNbcKkNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=13 ) .. [#constitutional-peasants-sketch] "Constitutional Peasants" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=JvKIWjnEPNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=14 ) .. [#ni-sketch] "Knights Who Say Ni" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=zIV4poUZAQo&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=15 ) .. [#homu] Homu (http://homu.io/) .. [#zuul] Zuul (http://docs.openstack.org/infra/zuul/) .. [#travis] Travis (https://travis-ci.org/) .. [#codeship] Codeship (https://codeship.com/) .. [#coverage] coverage.py (https://pypi.python.org/pypi/coverage) .. [#coveralls] Coveralls (https://coveralls.io/) .. [#codecov] Codecov (https://codecov.io/) .. [#pypatcher] Pypatcher (https://github.com/kushaldas/pypatcher) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Sun Jan 17 18:20:35 2016 From: phd at phdru.name (Oleg Broytman) Date: Mon, 18 Jan 2016 00:20:35 +0100 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: <20160117232035.GA2589@phdru.name> Hi! On Sun, Jan 17, 2016 at 08:48:42PM +0000, Brett Cannon wrote: > Change sys._mercurial > ''''''''''''''''''''' > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` > attribute will need to be removed. An equivalent ``sys._git`` > attribute will be needed to take its place. Shouldn't there be a sys._vcs with possible values 'subversion', 'mercurial' or 'git'? Well, currently it can be only 'git', of course, but in the future it will change. > Rejected Ideas > ============== > Commit multi-release changes in bugfix branch first > --------------------------------------------------- > As the current development process has changes committed in the > oldest branch first and then merged up to the default branch, the > question came up as to whether this workflow should be perpetuated. > In the end it was decided that committing in the newest branch and > then cherrypicking changes into older branches would work best as That's a rather strange workflow. Gitworkflows recommands against it [1], people recommends against it [2], [3]. > most people will instinctively work off the newest branch and it is a > more common workflow when using Git [#git]_. Most people commit to master because most [visible] repositories are for web sites/services/applications that don't have many branches. But for a project with a lot of release branches merge workflow is usually recommended. 1. https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html 2. https://stackoverflow.com/a/1241829 3. http://dan.bravender.net/2011/10/20/Why_cherry-picking_should_not_be_part_of_a_normal_git_workflow.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ezio.melotti at gmail.com Sun Jan 17 19:30:24 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Mon, 18 Jan 2016 02:30:24 +0200 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On Sun, Jan 17, 2016 at 10:48 PM, Brett Cannon wrote: > Here is the first complete draft of the PEP laying out what needs to happen > to migrate to GitHub (along with some extras that would be nice). The PEP > can also be viewed at > https://github.com/brettcannon/github-transition-pep/blob/master/pep-0512.rst > or at https://www.python.org/dev/peps/ once the PEP index is updated. > > My hope is that I captured all of the requirements for the various > repositories. If we can agree that is the case then we can start work on the > various requirements and start the migration work! > > > ---------- > PEP: 512 > Title: Migrating from hg.python.org to GitHub > Version: $Revision$ > Last-Modified: $Date$ > Author: Brett Cannon > Status: Active > Type: Process > Content-Type: text/x-rst > Created: > Post-History: 17-Jan-2015 > > Abstract > ======== > This PEP outlines the steps required to migrate Python's development > process from Mercurial [#hg]_ as hosted at > hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting > the minimum goals of this PEP should allow for the development > process of Python to be as productive as it currently is, and meeting > its extended goals should improve it. > > Rationale > ========= > In 2014, it became obvious that Python's custom development > process was becoming a hindrance. As an example, for an external > contributor to submit a fix for a bug that eventually was committed, > the basic steps were: > > 1. Open an issue for the bug at bugs.python.org [#b.p.o]_. > 2. Checkout out the CPython source code from hg.python.org [#h.p.o]_. > 3. Make the fix. > 4. Upload a patch. > 5. Have a core developer review the patch using our fork of the > Rietveld code review tool [#rietveld]_. > 6. Download the patch to make sure it still applies cleanly. > 7. Run the test suite manually. > 8. Commit the change manually. > 9. If the change was for a bugfix release, merge into the > in-development branch. > 10. Run the test suite manually again. > 11. Commit the merge. > 12. Push the changes. > > This is a very heavy, manual process for core developers. Even in the > simple case, you could only possibly skip the code review step, as you > would still need to build the documentation. This led to patches > languishing on the issue tracker due to core developers not being > able to work through the backlog fast enough to keep up with > submissions. In turn, that led to a side-effect issue of discouraging > outside contribution due to frustration from lack of attention, which > is dangerous problem for an open source project as it runs counter to > having a viable future for the project. Simply accepting patches > uploaded to bugs.python.org [#b.p.o]_ is potentially simple for an > external contributor, it is as slow and burdensome as it gets for > a core developer to work with. > > Hence the decision was made in late 2014 that a move to a new > development process was needed. A request for PEPs > proposing new workflows was made, in the end leading to two: > PEP 481 and PEP 507 proposing GitHub [#github]_ and > GitLab [#gitlab]_, respectively. > > The year 2015 was spent off-and-on working on those proposals and > trying to tease out details of what made them different from each > other on the core-workflow mailing list [#core-workflow]_. > PyCon US 2015 also showed that the community was a bit frustrated > with our process due to both cognitive overhead for new contributors > and how long it was taking for core developers to > look at a patch (see the end of Guido van Rossum's > keynote at PyCon US 2015 [#guido-keynote]_ as an example of the > frustration). > > On January 1, 2016, the decision was made by Brett Cannon to move the > development process to GitHub. The key reasons for choosing GitHub > were [#reasons]_: > > * Maintaining custom infrastructure has been a burden on volunteers > (e.g., a custom fork of Rietveld [#rietveld]_ > that is not being maintained is currently being used). > * The custom workflow is very time-consuming for core developers > (not enough automated tooling built to help support it). > * The custom workflow is a hindrance to external contributors > (acts as a barrier of entry due to time required to ramp up on > development process). > * There is no feature differentiating GitLab from GitHub beyond > GitLab being open source. > * Familiarity with GitHub is far higher amongst core developers and > external contributors than with GitLab. > * Our BDFL prefers GitHub (who would be the first person to tell > you that his opinion shouldn't matter, but the person making the > decision felt it was important that the BDFL feel comfortable with > the workflow of his own programming language to encourage his > continued participation). > > There's even already an unofficial image to use to represent the > migration to GitHub [#pythocat]_. > > The overarching goal of this migration is to improve the development > process to the extent that a core developer can go from external > contribution submission through all the steps leading to committing > said contribution all from within a browser on a tablet with WiFi. > All of this will be done in such a way that if an external > contributor chooses not to use GitHub then they will continue to have > that option. > > Repositories to Migrate > ======================= > While hg.python.org [#h.p.o]_ hosts many repositories, there are only > six key repositories that must move: > > 1. devinabox [#devinabox-repo]_ > 2. benchmarks [#benchmarks-repo]_ > 3. tracker [#tracker-repo]_ I don't think the tracker repo /must/ move (at least not immediately). The tracker repo includes 7 subrepos (our own forks of Roundup, Rietveld, and django-gae2django, and the instances for b.p.o, the meta tracker, the Jython tracker, and the setuptools tracker) and migrating it now means that: 1) The total number of repos to be converted will be 12* instead of 5; 2) I (and RDM/MvL) will have to learn a new workflow while updating the tracker for the migration; * 10 if we exclude Rietveld and gae2django (but that means holding up the migration until they are removed) I think it would be best to: 1) migrate the other repos; 2) work on the b.p.o/github integration using HG; 3) remove Rietveld; 4) (possibly) migrate the 4 tracker instances to GitHub Our Roundup fork is also an HG clone of upstream Roundup (with a separate branch with our changes), so migrating that to git, while doable, might be problematic. FWIW when we switched from SVN to HG it took over a year and half before I migrated the tracker repos, and I don't remember anyone being particularly concerned that we were still using SVN or relieved when we eventually moved it to HG. Personally I wouldn't mind if we kept them on HG (RDM/MvL might disagree though), but if it's a burden for the infra-team to keep hg.python.org alive, I would prefer if we moved them later rather than sooner. > 4. peps [#peps-repo]_ > 5. devguide [#devguide-repo]_ > 6. cpython [#cpython-repo]_ > > The devinabox, benchmarksm and tracker repositories are code-only. Typo: benchmarks > The peps and devguide repositories involve the generation of webpages. > And the cpython repository has special requirements for integration > with bugs.python.org [#b.p.o]_. > > Migration Plan > ============== > The migration plan is separated into sections based on what is > required to migrate the repositories listed in the > `Repositories to Migrate`_ section. Completion of requirements > outlined in each section should unblock the migration of the related > repositories. The sections are expected to be completed in order, but > not necessarily the requirements within a section. > > Requirements for Code-Only Repositories > --------------------------------------- > Completion of the requirements in this section will allow the > devinabox, benchmarks, and tracker repositories to move to > GitHub. While devinabox has a sufficiently descriptive name, the > benchmarks repository does not; therefore, it will be named > "python-benchmark-suite". The tracker repo also has a misleading name > and will be renamed "bugs.python.org". As mentioned above, tracker contains 7 subrepos, and the bugs.python.org instance is tracker/python-dev. Roundup (tracker/roundup) is running 4 tracker instances (tracker/python-dev, tracker/meta for the metatracker, tracker/jython for the Jython tracker, and tracker/setuptools for https://bugs.python.org/setuptools/ (now hosted on BitBucket but kept around for historical reasons)). > > Create a 'python-dev' team > '''''''''''''''''''''''''' > To manage permissions, a 'python-dev' team will be created as part of > the python organization [#github-python-org]_. Any repository that is > moved will have the 'python-dev' team added to it with write > permissions [#github-org-perms]_. Anyone who previously had rights to > manage SSH keys on hg.python.org will become a team maintainer for the > 'python-dev' team. > > Define commands to move a Mercurial repository to Git > ''''''''''''''''''''''''''''''''''''''''''''''''''''' > Since moving to GitHub also entails moving to Git [#git]_, we must > decide what tools and commands we will run to translate a Mercurial > repository to Git. The exact tools and steps to use are an > open issue; see `Tools and commands to move from Mercurial to Git`_. > > CLA enforcement > ''''''''''''''' > A key part of any open source project is making sure that its source > code can be properly licensed. This requires making sure all people > making contributions have signed a contributor license agreement > (CLA) [#cla]_. Up until now, enforcement of CLA signing of > contributed code has been enforced by core developers checking > whether someone had an ``*`` by their username on > bugs.python.org [#b.p.o]_. With this migration, the plan is to start > off with automated checking and enforcement of contributors signing > the CLA. > > Adding GitHub username support to bugs.python.org > +++++++++++++++++++++++++++++++++++++++++++++++++ > To keep tracking of CLA signing under the direct control of the PSF, > tracking who has signed the PSF CLA will be continued by marking that > fact as part of someone's bugs.python.org user profile. What this > means is that an association will be needed between a person's > bugs.python.org [#b.p.o]_ account and their GitHub account, which > will be done through a new field in a user's profile. > We have to decide how to deal with users that don't have a b.p.o account. The two options that we discussed in the previous mails are: 1) require them to create a b.p.o account; 2) allow them to log in to b.p.o using their github account; (see also next section) > A bot to enforce CLA signing > ++++++++++++++++++++++++++++ > With an association between someone's GitHub account and their > bugs.python.org [#b.p.o]_ account, which has the data as to whether > someone has signed the CLA, a bot can monitor pull requests on > GitHub and denote whether the contributor has signed the CLA. > > If the user has signed the CLA, the bot will add a positive label to > the issue to denote the pull request has no CLA issues (e.g., a green > label stating, "CLA: ?"). If the contributor has not signed a CLA, > a negative label will be added to the pull request will be blocked > using GitHub's status API (e.g., a red label stating, "CLA: ?"). If a > contributor lacks a bugs.python.org account, that will lead to > another label (e.g., "CLA: ? (no account)"). Using a label for both > positive and negative cases provides a fallback notification if the > bot happens to fail, preventing potential false-positives or > false-negatives. It also allows for an easy way to trigger the bot > again by simply removing a CLA-related label. > > Requirements for Web-Related Repositories > ----------------------------------------- > Due to their use for generating webpages, the > devguide [#devguide-repo]_ and peps [#peps-repo]_ repositories need > their respective processes updated to pull from their new Git > repositories. > > The devguide repository might also need to be named > ``python-devguide`` to make sure the repository is not ambiguous > when viewed in isolation from the > python organization [#github-python-org]_. > > Requirements for the cpython Repository > --------------------------------------- > Obviously the most active and important repository currently hosted > at hg.python.org [#h.p.o]_ is the cpython > repository [#cpython-repo]_. Because of its importance and high- > frequency use, it requires more tooling before being moved to GitHub > compared to the other repositories mentioned in this PEP. > > Document steps to commit a pull request > ''''''''''''''''''''''''''''''''''''''' > During the process of choosing a new development workflow, it was > decided that a linear history is desired. This means that the > convenient "Merge" button in GitHub pull requests is undesireable, as Typo: undesirable > it creates a merge commit along with all of the contributor's > individual commits (this does not affect the other repositories where > the desire for a linear history doesn't exist). > > Luckily, Git [#git]_ does not require GitHub's workflow and so one can > be chosen which gives us a linear history by using Git's CLI. The > expectation is that all pull requests will be fast-forwarded and > rebased before being pushed to the master repository. This should > give proper attribution to the pull request author in the Git > history. > > A second set of recommended commands will also be written for > committing a contribution from a patch file uploaded to > bugs.python.org [#b.p.o]_. This will obviously help keep the linear > history, but it will need to be made to have attribution to the patch > author. > > The exact sequence of commands that will be given as guidelines to > core developers is an open issue: > `Git CLI commands for committing a pull request to cpython`_. > > Handling Misc/NEWS > '''''''''''''''''' > Traditionally the ``Misc/NEWS`` file [#news-file]_ has been problematic > for changes which spanned Python releases. Often times there will be > merge conflicts when committing a change between e.g., 3.5 and 3.6 > only in the ``Misc/NEWS`` file. It's so common, in fact, that the > example instructions in the devguide explicitly mention how to > resolve conflicts in the ``Misc/NEWS`` file > [#devguide-merge-across-branches]_. As part of our tool > modernization, working with the ``Misc/NEWS`` file will be > simplified. > > There are currently two competing approaches to solving the > ``Misc/NEWS`` problem which are discussed in an open issue: > `How to handle the Misc/NEWS file`_. > > Handling Misc/ACKS > '''''''''''''''''' > Traditionally the ``Misc/ACKS`` file [#acks-file]_ has been managed > by hand. But thanks to Git supporting an ``author`` value as well as > a ``committer`` value per commit, authorship of a commit can be part > of the history of the code itself. > > As such, manual management of ``Misc/ACKS`` will become optional. A > script will be written that will collect all author and committer > names and merge them into ``Misc/ACKS`` with all of the names listed > prior to the move to Git. Running this script will become part of the > release process. > > Linking pull requests to issues > ''''''''''''''''''''''''''''''' > Historically, external contributions were attached to an issue on > bugs.python.org [#b.p.o]_ thanks to the fact that all external > contributions were uploaded as a file. For changes committed by a > core developer who committed a change directly, the specifying of an > issue number in the commit message of the format ``Issue #`` at the > start of the message led to a comment being posted to the issue > linking to the commit. > > Linking a pull request to an issue > ++++++++++++++++++++++++++++++++++ > An association between a pull request and an issue is needed to track > when a fix has been proposed. The association needs to be many-to-one > as there can take multiple pull requests to solve a single issue > (technically it should be a many-to-many association for when a > single fix solves multiple issues, but this is fairly rare and issues > can be merged into one using the ``Superceder`` field on the issue > tracker). > > Association between a pull request and an issue will be done based on > detecting the regular expression``[Ii]ssue #(?P\d+)``. If > this is specified in either the title or in the body of a message on > a pull request then connection will be made on > bugs.python.org [#b.p.o]_. A label will also be added to the pull > request to signify that the connection was made successfully. This > could lead to incorrect associations if the wrong issue or > referencing another issue was done, but these are rare occasions. > Is there a way to associate the PR to an issue (in case the user forgot) or change association (in case it got the wrong number) after the creation of the PR? > Notify the issue if the pull request is committed > +++++++++++++++++++++++++++++++++++++++++++++++++ > Once a pull request is closed (merged or not), the issue should be > updated to reflect this fact. > > Update linking service for mapping commit IDs to URLs > ''''''''''''''''''''''''''''''''''''''''''''''''''''' > Currently you can use https://hg.python.org/lookup/ with a revision > ID from either the Subversion or Mercurial copies of the > cpython repo [#cpython-repo]_ to get redirected to the URL for that > revision in the Mercurial repository. The URL rewriter will need to > be updated to redirect to the Git repository and to support the new > revision IDs created for the Git repository. > > Create https://git.python.org > ''''''''''''''''''''''''''''' > Just as hg.python.org [#h.p.o]_ currently points to the Mercurial > repository for Python, git.python.org should do the equivalent for > the Git repository. > Is this simply a redirect to GitHub or something else? We might also want to encourage the use of git.python.org in the Devguide and elsewhere to make future migrations simpler. > Backup of pull request data > ''''''''''''''''''''''''''' > Since GitHub [#github]_ is going to be used for code hosting and code > review, those two things need to be backed up. In the case of code > hosting, the backup is implicit as all non-shallow Git [#git]_ clones > contain the full history of the repository, hence there will be many > backups of the repository. > If possible I would still prefer an "official" backup. I don't think we want to go around asking who has the latest copy of the repo in case something happens to GitHub. > The code review history does not have the same implicit backup > mechanism as the repository itself. That means a daily backup of code > review history should be done so that it is not lost in case of any > issues with GitHub. It also helps guarantee that a migration from > GitHub to some other code review system is feasible were GitHub to > disappear overnight. > Is there an API to export this or do we need to write another bot and decide how to collect/store the reviews? > Change sys._mercurial > ''''''''''''''''''''' > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` > attribute will need to be removed. An equivalent ``sys._git`` > attribute will be needed to take its place. > > Optional, Planned Features > -------------------------- > Once the cpython repository [#cpython-repo]_ is migrated, all > repositories will have been moved to GitHub [#github]_ and the > development process should be on equal footing as before. But a key > reason for this migration is to improve the development process, > making it better than it has ever been. This section outlines some > plans on how to improve things. > > It should be mentioned that overall feature planning for > bugs.python.org [#b.p.o]_ -- which includes plans independent of this > migration -- are tracked on their own wiki page [#tracker-plans]_. > > Bot to handle pull request merging > '''''''''''''''''''''''''''''''''' > As stated in the section entitled > "`Document steps to commit a pull request`_", the desire is to > maintain a linear history for cpython. Unfortunately, > Github's [#github]_ web-based workflow does not support a linear > history. Because of this, a bot should be written to substitute for > GitHub's in-browser commit abilities. > > To start, the bot should accept commands to commit a pull request > against a list of branches. This allows for committing a pull request > that fixes a bug in multiple versions of Python. > > More advanced features such as a commit queue can come later. This > would linearly apply accepted pull requests and verify that the > commits did not interfere with each other by running the test suite > and backing out commits if the test run failed. To help facilitate > the speed of testing, all patches committed since the last test run > can be applied and run in a single test run as the optimistic > assumption is that the patches will work in tandem. > > Inspiration or basis of the bot could be taken from pre-existig bots > such as Homu [#homu]_ or Zuul [#zuul]_. > > The name given to this bot in order to give it commands is an open > issue: `Naming the commit bot`_. > > Continuous integration per pull request > ''''''''''''''''''''''''''''''''''''''' > To help speed up pull request approvals, continuous integration > testing should be used. This helps mitigate the need for a core > developer to download a patch simply to run the test suite against > the patch. > > Which free CI service to use is an open issue: > `Choosing a CI service`_. > > Test coverage report > '''''''''''''''''''' > Getting an up-to-date test coverage report for Python's standard > library would be extremely beneficial as generating such a report can > take quite a while to produce. > > There are a couple pre-existing services that provide free test > coverage for open source projects. Which option is best is an open > issue: `Choosing a test coverage service`_. > Do we want to eventually request that all new code introduced is fully covered by tests? I think having an indication of how much code in a PR is covered by tests would be useful regardless of the answer to the previous question. > Notifying issues of pull request comments > ''''''''''''''''''''''''''''''''''''''''' > The current development process does not include notifying an issue > on bugs.python.org [#b.p.o]_ when a review comment is left on > Rietveld [#rietveld]_. It would be nice to fix this so that people > can subscribe only to comments at bugs.python.org and not > GitHub [#github]_ and yet still know when something occurs on GitHub > in terms of review comments on relevant pull requests. Current > thinking is to post a comment to bugs.python.org to the relevant > issue when at least one review comment has been made over a certain > period of time (e.g., 15 or 30 minutes). This keeps the email volume > down for those that receive both GitHub and bugs.python.org email > notifications while still making sure that those only following > bugs.python.org know when there might be a review comment to address. > > Allow bugs.python.org to use GitHub as a login provider > ''''''''''''''''''''''''''''''''''''''''''''''''''''''' > As of right now, bugs.python.org [#b.p.o]_ allows people to log in > using Google, Launchpad, or OpenID credentials. It would be good to > expand this to GitHub credentials. > > Web hooks for re-generating web content > ''''''''''''''''''''''''''''''''''''''' > The content at https://docs.python.org/, > https://docs.python.org/devguide, and > https://www.python.org/dev/peps/ are all derived from files kept in > one of the repositories to be moved as part of this migration. As > such, it would be nice to set up appropriate webhooks to trigger > rebuilding the appropriate web content when the files they are based > on change instead of having to wait for, e.g., a cronjob to trigger. > > Link web content back to files that it is generated from > '''''''''''''''''''''''''''''''''''''''''''''''''''''''' > It would be helpful for people who find issues with any of the > documentation that is generated from a file to have a link on each > page which points back to the file on GitHub [#github]_ that stores > the content of the page. That would allow for quick pull requests to > fix simple things such as spelling mistakes. > Here you are talking about PEPs/devguide/docs.p.o, right? FWIW this was problem was supposed to be fixed with pootle, but that project seems dead (not sure if due to technical reasons, or simply because no one had time to integrate it). > Splitting out parts of the documentation into their own repositories > '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' > While certain parts of the documentation at https://docs.python.org > change with the code, other parts are fairly static and are not > tightly bound to the CPython code itself. The following sections of > the documentation fit this category of slow-changing, > loosely-coupled: > > * `Tutorial `__ > * `Python Setup and Usage `__ > * `HOWTOs `__ > * `Installing Python Modules > `__ > * `Distributing Python Modules > `__ > * `Extending and Embedding > `__ > * `FAQs `__ > > These parts of the documentation could be broken out into their own > repositories to simplify their maintenance and to expand who has > commit rights to them to ease in their maintenance. I would still consider these somewhat dynamic (especially between Py 2 and Py 3). There are other documents that are more version-independent, such as the whatsnew pages, the meta information section at the bottom of docs.p.o/index (that includes pages such as https://docs.python.org/3/bugs.html ) and a new page with the current status of the releases proposed in http://bugs.python.org/issue25296 > > Status > ====== > Requirements for migrating the devinabox [#devinabox-repo]_, > benchmarks [#benchmarks-repo]_, and tracker [#tracker-repo]_ > repositories: > > * Not started > > - `Create a 'python-dev' team`_ > - `Define commands to move a Mercurial repository to Git`_ > - `Adding GitHub username support to bugs.python.org`_ > - `A bot to enforce CLA signing`_ > > * In progress > > - None > > * Completed > > - None > > Repositories whose build steps need updating: > > * Not started > > - peps [#peps-repo]_ > - devguide [#devguide-repo]_ > > * In progress > > - None > > * Completed > > - None > > Requirements to move over the cpython repo [#cpython-repo]_: > > * Not started > > - `Document steps to commit a pull request`_ > - `Handling Misc/NEWS`_ > - `Handling Misc/ACKS`_ > - `Linking a pull request to an issue`_ > - `Notify the issue if the pull request is committed`_ > - `Update linking service for mapping commit IDs to URLs`_ > - `Create https://git.python.org`_ > - `Backup of pull request data`_ > - `Change sys._mercurial`_ > > * In progress > > - None > > * Completed > > - None > > Optional features: > > * Not started > > - `Bot to handle pull request merging`_ > - `Continuous integration per pull request`_ > - `Test coverage report`_ > - `Notifying issues of pull request comments`_ > - `Allow bugs.python.org to use GitHub as a login provider`_ > - `Web hooks for re-generating web content`_ > - `Link web content back to files that it is generated from`_ > - `Splitting out parts of the documentation into their own repositories`_ > > * In progress > > - None > > * Completed > > - None > > Open Issues > =========== > For this PEP, open issues are ones where a decision needs to be made > to how to approach or solve a problem. Open issues do not entail > coordination issues such as who is going to write a certain bit of > code. > > The fate of hg.python.org > ------------------------- > With the code repositories moving over to Git [#git]_, there is no > technical need to keep hg.python.org [#h.p.o]_ running. Having said > that, some in the community would like to have it stay functioning as > a Mercurial [#hg]_ mirror of the Git repositories. Others have said > that they still want a mirror, but one using Git. > > As maintaining hg.python.org is not necessary, it will be up to the > PSF infrastructure committee to decide if they want to spend the > time and resources to keep it running. They may also choose whether > they want to host a Git mirror on PSF infrastructure. > +1 to keep a read-only mirror. Also see above about the tracker repo and our Roundup fork. > Tools and commands to move from Mercurial to Git > ------------------------------------------------ > A decision needs to be made on exactly what tooling and what commands > involving those tools will be used to convert a Mercurial repository > to Git. Currently a suggestion has been made to use > https://github.com/frej/fast-export. Another suggestion is to use > https://github.com/felipec/git-remote-hg. Finally, > http://hg-git.github.io/ has been suggested. > > Git CLI commands for committing a pull request to cpython > --------------------------------------------------------- > Because Git [#git]_ may be a new version control system for core > developers, the commands people are expected to run will need to be > written down. These commands also need to keep a linear history while > giving proper attribution to the pull request author. > > Another set of commands will also be necessary for when working with > a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear > history will be kept implicitly, but it will need to make sure to > keep/add attribution. > Nick Coghlan, Pierre-Yves David (a Mercurial dev), and Shiyao Ma (one of our GSoC student) have been working on an HG extension that simplifies interaction with the bug tracker (see the list of patches, download/apply them, upload new patches): https://bitbucket.org/introom/hg-cpydev In a previous email, someone mentioned an alias that allows an easier interaction with PRs. Would it make sense to write and distribute an official git extension that provides extra commands/aliases for these set of commands? (I guess the answer depends on how many tasks we have and how straightforward it is to do with plain git commands.) > How to handle the Misc/NEWS file > -------------------------------- > There are two competing approaches to handling > ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on > bugs.python.org [#b.p.o]_. This would mean an issue that is marked > as "resolved" could not be closed until a news entry is added in the > "news" field in the issue tracker. The benefit of tying the news > entry to the issue is it makes sure that all changes worthy of a news > entry have an accompanying issue. It also makes classifying a news > entry automatic thanks to the Component field of the issue. The > Versions field of the issue also ties the news entry to which Python > releases were affected. A script would be written to query > bugs.python.org for relevant new entries for a release and to produce > the output needed to be checked into the code repository. This > approach is agnostic to whether a commit was done by CLI or bot. > > The competing approach is to use an individual file per news entry, > containg the text for the entry. In this scenario each feature Typo: containing > release would have its own directory for news entries and a separate > file would be created in that directory that was either named after > the issue it closed or a timestamp value (which prevents collisions). > Merges across branches would have no issue as the news entry file > would still be uniqeuely named and in the directory of the latest Typo: uniquely > version that contained the fix. A script would collect all news entry > files no matter what directory they reside in and create an > appropriate news file (the release directory can be ignored as the > mere fact that the file exists is enough to represent that the entry > belongs to the release). Classification can either be done by keyword > in the new entry file itself or by using subdirectories representing > each news entry classification in each release directory (or > classification of news entries could be dropped since critical > information is captured by the "What's New" documents which are > organized). The benefit of this approach is that it keeps the changes > with the code that was actually changed. It also ties the message to > being part of the commit which introduced the change. For a commit > made through the CLI, a script will be provided to help generate the > file. In a bot-driven scenario, the merge bot will have a way to > specify a specific news entry and create the file as part of its > flattened commit (while most likely also supporting using the first > line of the commit message if no specific news entry was specified). > Code for this approach has been written previously for the Mercurial > workflow at http://bugs.python.org/issue18967. There is also a tool > from the community at https://pypi.python.org/pypi/towncrier. > Does using git (and fast-forward merges, rebases, etc.) still create conflicts? (I guess the answer is yes, but perhaps we should double-check.) If it does, there's also a third option: writing a merge script. I wrote a basic one for hg and it seemed to work decently, perhaps with git it's even easier. > Naming the commit bot > --------------------- > As naming things can lead to bikeshedding of epic proportions, Brett > Cannon will choose the final name of the commit bot (the name of the > project for the bot itself can be anything, this is purely for the > name used in giving commands to the bot). The name will come from > Monty Python, which is only fitting since Python is named after the > comedy troupe. It will most likely come from > 'Monty Python and the Holy Grail' [#holy-grail]_ (which happens to be > how Brett was introduced to Monty Python). Current ideas on the name > include: > > "Black Knight" sketch [#black-knight-sketch]_: > > * black-knight > * none-shall-pass > * just-a-flesh-wound > > "Bridge of Death" sketch [#bridge-of-death-sketch]_: > > * bridge-keeper > * man-from-scene-24 > * five-questions > * what-is-your-quest > * blue-no-green > * air-speed-velocity > * your-favourite-colour > (and that specific spelling; Monty Python is British, after all) > > "Killer rabbit" sketch [#killer-rabbit-sketch]_: > > * killer-rabbit > * holy-hand-grenade > * 5-is-right-out > > "Witch Village" sketch [#witch-village-sketch]_: > > * made-of-wood > * burn-her > > "French Taunter" sketch [#french-taunter-sketch]_: > > * elderberries > * kanigget > > "Constitutional Peasants" sketch [#constitutional-peasants-sketch]_: > > * dennis > * from-the-masses > * watery-tart > > "Knights Who Say Ni" sketch [#ni-sketch]_: > > * shubbery > * ni > > From "Monty Python and the Holy Grail" in general: > > * brave-sir-robin > FWIW the bot that currently reports HG commits on IRC is called deadparrot. Best Regards, Ezio Melotti > Choosing a CI service > --------------------- > There are various CI services that provide free support for open > source projects hosted on GitHub [#github]_. Two such examples are > Travis [#travis]_ and Codeship [#codeship]_. Whatever solution is > chosen will need to not time out in the time it takes to execute > Python's test suite. It should optimally provide access to multiple C > compilers for more thorough testing. Network access is also > beneficial. > > The current CI service for Python is Pypatcher [#pypatcher]_. A > request can be made in IRC to try a patch from > bugs.python.org [#b.p.o]_. The results can be viewed at > https://ci.centos.org/job/cPython-build-patch/ . > > Choosing a test coverage service > -------------------------------- > Getting basic test coverage of Python's standard library can be > created simply by using coverage.py [#coverage]_. Getting > thorough test coverage is actually quite tricky, with the details > outlined in the devinabox's README [#devinabox-repo]_. It would be > best if a service could be found that would allow for thorough test > coverage, but it might not be feasible. > > Free test coverage services include Coveralls [#coveralls]_ and > Codecov [#codecov]_. > > Rejected Ideas > ============== > Separate Python 2 and Python 3 repositories > ------------------------------------------- > It was discussed whether separate repositories for Python 2 and > Python 3 were desired. The thinking was that this would shrink the > overall repository size which benefits people with slow Internet > connections or small bandwidth caps. > > In the end it was decided that it was easier logistically to simply > keep all of CPython's history in a single repository. > > Commit multi-release changes in bugfix branch first > --------------------------------------------------- > As the current development process has changes committed in the > oldest branch first and then merged up to the default branch, the > question came up as to whether this workflow should be perpetuated. > In the end it was decided that committing in the newest branch and > then cherrypicking changes into older branches would work best as > most people will instinctively work off the newest branch and it is a > more common workflow when using Git [#git]_. > > Deriving ``Misc/NEWS`` from the commit logs > ------------------------------------------- > As part of the discussion surrounding `Handling Misc/NEWS`_, the > suggestion has come up of deriving the file from the commit logs > itself. In this scenario, the first line of a commit message would be > taken to represent the news entry for the change. Some heuristic to > tie in whether a change warranted a news entry would be used, e.g., > whether an issue number is listed. > > This idea has been rejected due to some core developers preferring to > write a news entry separate from the commit message. The argument is > the first line of a commit message compared to that of a news entry > have different requirements in terms of brevity, what should be said, > etc. > > References > ========== > .. [#h.p.o] https://hg.python.org > > .. [#GitHub] GitHub (https://github.com) > > .. [#hg] Mercurial (https://www.mercurial-scm.org/) > > .. [#git] Git (https://git-scm.com/) > > .. [#b.p.o] https://bugs.python.org > > .. [#rietveld] Rietveld (https://github.com/rietveld-codereview/rietveld) > > .. [#gitlab] GitLab (https://about.gitlab.com/) > > .. [#core-workflow] core-workflow mailing list > (https://mail.python.org/mailman/listinfo/core-workflow) > > .. [#guido-keynote] Guido van Rossum's keynote at PyCon US > (https://www.youtube.com/watch?v=G-uKNd5TSBw) > > .. [#reasons] Email to core-workflow outlining reasons why GitHub was > selected > > (https://mail.python.org/pipermail/core-workflow/2016-January/000345.html) > > .. [#benchmarks-repo] Mercurial repository for the Unified Benchmark Suite > (https://hg.python.org/benchmarks/) > > .. [#devinabox-repo] Mercurial repository for devinabox > (https://hg.python.org/devinabox/) > > .. [#peps-repo] Mercurial repository of the Python Enhancement Proposals > (https://hg.python.org/peps/) > > .. [#tracker-repo] bugs.python.org code repository > (https://hg.python.org/tracker/python-dev/) > > .. [#devguide-repo] Mercurial repository for the Python Developer's Guide > (https://hg.python.org/devguide/) > > .. [#cpython-repo] Mercurial repository for CPython > (https://hg.python.org/cpython/) > > .. [#github-python-org] Python organization on GitHub > (https://github.com/python) > > .. [#github-org-perms] GitHub repository permission levels > > (https://help.github.com/enterprise/2.4/user/articles/repository-permission-levels-for-an-organization/) > > .. [#cla] Python Software Foundation Contributor Agreement > (https://www.python.org/psf/contrib/contrib-form/) > > .. [#news-file] ``Misc/NEWS`` > (https://hg.python.org/cpython/file/default/Misc/NEWS) > > .. [#acks-file] ``Misc/ACKS`` > (https://hg.python.org/cpython/file/default/Misc/ACKS) > > .. [#devguide-merge-across-branches] Devguide instructions on how to merge > across branches > > (https://docs.python.org/devguide/committing.html#merging-between-different-branches-within-the-same-major-version) > > .. [#pythocat] Pythocat (https://octodex.github.com/pythocat/) > > .. [#tracker-plans] Wiki page for bugs.python.org feature development > (https://wiki.python.org/moin/TrackerDevelopmentPlanning) > > .. [#black-knight-sketch] The "Black Knight" sketch from "Monty Python and > the Holy Grail" > (https://www.youtube.com/watch?v=dhRUe-gz690) > > .. [#bridge-of-death-sketch] The "Bridge of Death" sketch from "Monty Python > and the Holy Grail" > (https://www.youtube.com/watch?v=cV0tCphFMr8) > > .. [#holy-grail] "Monty Python and the Holy Grail" sketches > > (https://www.youtube.com/playlist?list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp) > > .. [#killer-rabbit-sketch] "Killer rabbit" sketch from "Monty Python and the > Holy Grail" > > (https://www.youtube.com/watch?v=Nvs5pqf-DMA&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=11) > > .. [#witch-village-sketch] "Witch Village" from "Monty Python and the Holy > Grail" > > (https://www.youtube.com/watch?v=k3jt5ibfRzw&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=12) > > .. [#french-taunter-sketch] "French Taunter" from "Monty Python and the Holy > Grail" > > (https://www.youtube.com/watch?v=A8yjNbcKkNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=13) > > .. [#constitutional-peasants-sketch] "Constitutional Peasants" from "Monty > Python and the Holy Grail" > > (https://www.youtube.com/watch?v=JvKIWjnEPNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=14) > > .. [#ni-sketch] "Knights Who Say Ni" from "Monty Python and the Holy Grail" > > (https://www.youtube.com/watch?v=zIV4poUZAQo&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=15) > > .. [#homu] Homu (http://homu.io/) > > .. [#zuul] Zuul (http://docs.openstack.org/infra/zuul/) > > .. [#travis] Travis (https://travis-ci.org/) > > .. [#codeship] Codeship (https://codeship.com/) > > .. [#coverage] coverage.py (https://pypi.python.org/pypi/coverage) > > .. [#coveralls] Coveralls (https://coveralls.io/) > > .. [#codecov] Codecov (https://codecov.io/) > > .. [#pypatcher] Pypatcher (https://github.com/kushaldas/pypatcher) > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct From vadmium+py at gmail.com Sun Jan 17 19:42:08 2016 From: vadmium+py at gmail.com (Martin Panter) Date: Mon, 18 Jan 2016 00:42:08 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: <20160117232035.GA2589@phdru.name> References: <20160117232035.GA2589@phdru.name> Message-ID: On 17 January 2016 at 23:20, Oleg Broytman wrote: > On Sun, Jan 17, 2016 at 08:48:42PM +0000, Brett Cannon wrote: >> Rejected Ideas >> ============== >> Commit multi-release changes in bugfix branch first >> --------------------------------------------------- >> As the current development process has changes committed in the >> oldest branch first and then merged up to the default branch, the >> question came up as to whether this workflow should be perpetuated. >> In the end it was decided that committing in the newest branch and >> then cherrypicking changes into older branches would work best as > > That's a rather strange workflow. Gitworkflows recommands against > it [1], people recommends against it [2], [3]. > >> most people will instinctively work off the newest branch and it is a >> more common workflow when using Git [#git]_. > > Most people commit to master because most [visible] repositories are > for web sites/services/applications that don't have many branches. But > for a project with a lot of release branches merge workflow is usually > recommended. > > 1. https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html > 2. https://stackoverflow.com/a/1241829 > 3. http://dan.bravender.net/2011/10/20/Why_cherry-picking_should_not_be_part_of_a_normal_git_workflow.html In theory I would also prefer sticking with the current process of merging the 3.x branches, and only cherry-picking for the 2.7 branch. Are there any more details on this decision? When making a patch for review I currently do it off the ?default? branch, because then one patch will usually encompass all the changes needed (but a patch against 3.5 for instance may miss an API added in 3.6). But when actually making a commit I think it is better to commit first to 3.5 (for instance). For handling pull requests from others, I can see that having the pull request against master would be easier for the original contributor. On the other hand, if the contributor were able to pre-arrange all the merges between branches (including resolving conflicts), it might be help the person doing the final push. But maybe this is too complicated for all parties; I dunno. From vadmium+py at gmail.com Sun Jan 17 20:36:56 2016 From: vadmium+py at gmail.com (Martin Panter) Date: Mon, 18 Jan 2016 01:36:56 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On 17 January 2016 at 20:48, Brett Cannon wrote: > Rationale > ========= > . . . > The overarching goal of this migration is to improve the development > process to the extent that a core developer can go from external > contribution submission through all the steps leading to committing > said contribution all from within a browser on a tablet with WiFi. Later on, at least for the main ?cpython? repository, you mention maintaining a linear history, avoiding the automatic Git Hub merge button, and cherry-picking between branches. I guess you would need the commit bot to help do all this from a browser. If so, mention that, otherwise this gives the impression that people will Git Hub?s merge button, which seems to contradict later sections. > Document steps to commit a pull request > ''''''''''''''''''''''''''''''''''''''' > During the process of choosing a new development workflow, it was > decided that a linear history is desired. This means that the > convenient "Merge" button in GitHub pull requests is undesireable, as > it creates a merge commit along with all of the contributor's > individual commits (this does not affect the other repositories where > the desire for a linear history doesn't exist). Please clarify whether this linear history is just avoiding the suprious merge commits that Git Hub can add for each pull request (merging a pull request with its parent commit), or whether we are going to stop merging maintainence branches into master, and do cherry-picks instead. > Linking a pull request to an issue > ++++++++++++++++++++++++++++++++++ > . . . > (technically it should be a many-to-many association for when a > single fix solves multiple issues, but this is fairly rare and issues > can be merged into one using the ``Superceder`` field on the issue > tracker). Spelling: ``Super[s]eder`` > Status > ====== > . . . > Repositories whose build steps need updating: There is some stray indentation starting here which seems to be messing with the markup. > The fate of hg.python.org > ------------------------- > With the code repositories moving over to Git [#git]_, there is no > technical need to keep hg.python.org [#h.p.o]_ running. I presume this means other repositories will be migrated somewhere else too? For instance, I recently had to update the ?pythontestdotnet? repository. > Tools and commands to move from Mercurial to Git > ------------------------------------------------ > A decision needs to be made on exactly what tooling and what commands > involving those tools will be used to convert a Mercurial repository > to Git. Currently a suggestion has been made to use > https://github.com/frej/fast-export. Another suggestion is to use > https://github.com/felipec/git-remote-hg. Finally, > http://hg-git.github.io/ has been suggested. The migration to Git might be a good opportunity to clean up some of the older Subversion history. I would like to investigate this. In particular, rewriting svnmerge commits as proper Git merges, and restoring the missing history for merges and imports from feature branches. On the other hand, there is already a Git mirror which many seem to be already using. Perhaps it would be good to ensure the commit hashes used there are kept current. Which tool is used to update that mirror? Are there any deficiencies that would mean we need to use a different tool, possibly affecting the commit hashes? > Separate Python 2 and Python 3 repositories > ------------------------------------------- > It was discussed whether separate repositories for Python 2 and > Python 3 were desired. The thinking was that this would shrink the > overall repository size which benefits people with slow Internet > connections or small bandwidth caps. > > In the end it was decided that it was easier logistically to simply > keep all of CPython's history in a single repository. Agreed. If people really only want one of the branches, they can set Git to only download from that branch. And Git supports shallow clones (only download the most recent commit, or N commits deep), although some functions may not work in this mode (but apparently things have improved since I learnt about it). From brett at python.org Sun Jan 17 21:16:06 2016 From: brett at python.org (Brett Cannon) Date: Mon, 18 Jan 2016 02:16:06 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: <20160117232035.GA2589@phdru.name> References: <20160117232035.GA2589@phdru.name> Message-ID: On Sun, 17 Jan 2016 at 15:20 Oleg Broytman wrote: > Hi! > > On Sun, Jan 17, 2016 at 08:48:42PM +0000, Brett Cannon > wrote: > > Change sys._mercurial > > ''''''''''''''''''''' > > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` > > attribute will need to be removed. An equivalent ``sys._git`` > > attribute will be needed to take its place. > > Shouldn't there be a sys._vcs with possible values 'subversion', > 'mercurial' or 'git'? Well, currently it can be only 'git', of course, > but in the future it will change. > This was discussed already the first time the attribute was added for Subversion and the decision that a VCS-specific attribute was preferred. > > > Rejected Ideas > > ============== > > Commit multi-release changes in bugfix branch first > > --------------------------------------------------- > > As the current development process has changes committed in the > > oldest branch first and then merged up to the default branch, the > > question came up as to whether this workflow should be perpetuated. > > In the end it was decided that committing in the newest branch and > > then cherrypicking changes into older branches would work best as > > That's a rather strange workflow. Gitworkflows recommands against > it [1], people recommends against it [2], [3]. > > > most people will instinctively work off the newest branch and it is a > > more common workflow when using Git [#git]_. > > Most people commit to master because most [visible] repositories are > for web sites/services/applications that don't have many branches. But > for a project with a lot of release branches merge workflow is usually > recommended. > > 1. https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html > 2. https://stackoverflow.com/a/1241829 > 3. > http://dan.bravender.net/2011/10/20/Why_cherry-picking_should_not_be_part_of_a_normal_git_workflow.html I'm personally not going to argue about this and change the current proposal right now, but if people want to start a new thread to discuss whether merging up or cherry-picking down is the better workflow they can and if consensus can be reached I will change the PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jan 17 21:36:43 2016 From: brett at python.org (Brett Cannon) Date: Mon, 18 Jan 2016 02:36:43 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: <20160117232035.GA2589@phdru.name> Message-ID: On Sun, 17 Jan 2016 at 16:42 Martin Panter wrote: > On 17 January 2016 at 23:20, Oleg Broytman wrote: > > On Sun, Jan 17, 2016 at 08:48:42PM +0000, Brett Cannon > wrote: > >> Rejected Ideas > >> ============== > >> Commit multi-release changes in bugfix branch first > >> --------------------------------------------------- > >> As the current development process has changes committed in the > >> oldest branch first and then merged up to the default branch, the > >> question came up as to whether this workflow should be perpetuated. > >> In the end it was decided that committing in the newest branch and > >> then cherrypicking changes into older branches would work best as > > > > That's a rather strange workflow. Gitworkflows recommands against > > it [1], people recommends against it [2], [3]. > > > >> most people will instinctively work off the newest branch and it is a > >> more common workflow when using Git [#git]_. > > > > Most people commit to master because most [visible] repositories are > > for web sites/services/applications that don't have many branches. But > > for a project with a lot of release branches merge workflow is usually > > recommended. > > > > 1. https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html > > 2. https://stackoverflow.com/a/1241829 > > 3. > http://dan.bravender.net/2011/10/20/Why_cherry-picking_should_not_be_part_of_a_normal_git_workflow.html > > In theory I would also prefer sticking with the current process of > merging the 3.x branches, and only cherry-picking for the 2.7 branch. > Are there any more details on this decision? > Nothing beyond that was what people seemed to prefer when this was discussed back in November/December. > > When making a patch for review I currently do it off the ?default? > branch, because then one patch will usually encompass all the changes > needed (but a patch against 3.5 for instance may miss an API added in > 3.6). But when actually making a commit I think it is better to commit > first to 3.5 (for instance). > > For handling pull requests from others, I can see that having the pull > request against master would be easier for the original contributor. > On the other hand, if the contributor were able to pre-arrange all the > merges between branches (including resolving conflicts), it might be > help the person doing the final push. But maybe this is too > complicated for all parties; I dunno. > As I said to Ezio, if people want to start a new thread to discuss this can come back to me with a different recommendation then that's fine by me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jan 17 21:40:53 2016 From: brett at python.org (Brett Cannon) Date: Mon, 18 Jan 2016 02:40:53 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On Sun, 17 Jan 2016 at 17:37 Martin Panter wrote: > On 17 January 2016 at 20:48, Brett Cannon wrote: > > Rationale > > ========= > > . . . > > The overarching goal of this migration is to improve the development > > process to the extent that a core developer can go from external > > contribution submission through all the steps leading to committing > > said contribution all from within a browser on a tablet with WiFi. > > Later on, at least for the main ?cpython? repository, you mention > maintaining a linear history, avoiding the automatic Git Hub merge > button, and cherry-picking between branches. I guess you would need > the commit bot to help do all this from a browser. Yes. > If so, mention > that, otherwise this gives the impression that people will Git Hub?s > merge button, which seems to contradict later sections. > Well, you get it for some of the repositories, just not cpython. And I don't entirely agree that the sentence suggests GitHub's built-in browser workflow is the answer, just that *some* in-browser workflow is desired. > > > Document steps to commit a pull request > > ''''''''''''''''''''''''''''''''''''''' > > During the process of choosing a new development workflow, it was > > decided that a linear history is desired. This means that the > > convenient "Merge" button in GitHub pull requests is undesireable, as > > it creates a merge commit along with all of the contributor's > > individual commits (this does not affect the other repositories where > > the desire for a linear history doesn't exist). > > Please clarify whether this linear history is just avoiding the > suprious merge commits that Git Hub can add for each pull request > (merging a pull request with its parent commit), or whether we are > going to stop merging maintainence branches into master, and do > cherry-picks instead. Both is the current plan. > > > Linking a pull request to an issue > > ++++++++++++++++++++++++++++++++++ > > . . . > > (technically it should be a many-to-many association for when a > > single fix solves multiple issues, but this is fairly rare and issues > > can be merged into one using the ``Superceder`` field on the issue > > tracker). > > Spelling: ``Super[s]eder`` > > > Status > > ====== > > . . . > > Repositories whose build steps need updating: > > There is some stray indentation starting here which seems to be > messing with the markup. > > > The fate of hg.python.org > > ------------------------- > > With the code repositories moving over to Git [#git]_, there is no > > technical need to keep hg.python.org [#h.p.o]_ running. > > I presume this means other repositories will be migrated somewhere > else too? For instance, I recently had to update the > ?pythontestdotnet? repository. > The assumption is everything will move long-term instead of fragmenting between GitHub and hg.python.org. > > > Tools and commands to move from Mercurial to Git > > ------------------------------------------------ > > A decision needs to be made on exactly what tooling and what commands > > involving those tools will be used to convert a Mercurial repository > > to Git. Currently a suggestion has been made to use > > https://github.com/frej/fast-export. Another suggestion is to use > > https://github.com/felipec/git-remote-hg. Finally, > > http://hg-git.github.io/ has been suggested. > > The migration to Git might be a good opportunity to clean up some of > the older Subversion history. I would like to investigate this. In > particular, rewriting svnmerge commits as proper Git merges, and > restoring the missing history for merges and imports from feature > branches. > > On the other hand, there is already a Git mirror > which many seem to be already > using. Perhaps it would be good to ensure the commit hashes used there > are kept current. I'm not going to worry about that if the mirror was not done accurately. > Which tool is used to update that mirror? Are there > any deficiencies that would mean we need to use a different tool, > possibly affecting the commit hashes? > I don't know. Eli Bendersky set up that mirror and he has said previously that he just did the bare minimum to get it working and didn't worry too much about correctness, etc. -Brett > > > Separate Python 2 and Python 3 repositories > > ------------------------------------------- > > It was discussed whether separate repositories for Python 2 and > > Python 3 were desired. The thinking was that this would shrink the > > overall repository size which benefits people with slow Internet > > connections or small bandwidth caps. > > > > In the end it was decided that it was easier logistically to simply > > keep all of CPython's history in a single repository. > > Agreed. If people really only want one of the branches, they can set > Git to only download from that branch. And Git supports shallow clones > (only download the most recent commit, or N commits deep), although > some functions may not work in this mode (but apparently things have > improved since I learnt about it). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Jan 17 21:57:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 18 Jan 2016 13:57:56 +1100 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On Mon, Jan 18, 2016 at 7:48 AM, Brett Cannon wrote: > Luckily, Git [#git]_ does not require GitHub's workflow and so one can > be chosen which gives us a linear history by using Git's CLI. The > expectation is that all pull requests will be fast-forwarded and > rebased before being pushed to the master repository. This should > give proper attribution to the pull request author in the Git > history. > > A second set of recommended commands will also be written for > committing a contribution from a patch file uploaded to > bugs.python.org [#b.p.o]_. This will obviously help keep the linear > history, but it will need to be made to have attribution to the patch > author. Point to note: If you want to make use of git's separate author and committer records (so the author is the person who wrote the original patch, and the committer is the core dev who actually pushed it), you'll forfeit the identifiable hashes on commits. Every commit will have to be rebased (or amended, which is a short-hand form of rebase) to change its committer, which creates a brand new commit with a new hash. GitHub won't recognize the push as incorporating the original commit, and neither will people's tools elsewhere. IMO this is a worthwhile trade-off, but it is a cost, and may be worth mentioning in the PEP. It's no worse than the current state (where a Mercurial commit completely loses track of its original author, unless it's mentioned in the human-readable message), but it's perhaps not what people will be expecting. (Also, when it comes time to write up the actual commands for people, it'll need to be made clear that an actual fast-forward isn't acceptable, as it won't update the committer. Amending or rebasing MUST be done.) ChrisA From ezio.melotti at gmail.com Mon Jan 18 00:37:01 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Mon, 18 Jan 2016 07:37:01 +0200 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On Mon, Jan 18, 2016 at 4:33 AM, Brett Cannon wrote: > > > On Sun, 17 Jan 2016 at 16:30 Ezio Melotti wrote: >> >> > >> > Repositories to Migrate >> > ======================= >> > While hg.python.org [#h.p.o]_ hosts many repositories, there are only >> > six key repositories that must move: >> > >> > 1. devinabox [#devinabox-repo]_ >> > 2. benchmarks [#benchmarks-repo]_ >> > 3. tracker [#tracker-repo]_ >> >> I don't think the tracker repo /must/ move (at least not immediately). > > > I don't think so either, just that it should probably happen at some point, > especially if hg.python.org gets shut down. > >> >> The tracker repo includes 7 subrepos (our own forks of Roundup, >> Rietveld, and django-gae2django, > > > Is that being used by anyone? > gae2django is necessary to run Rietveld. If we remove Rietveld, we won't need gae2django anymore. Roundup is used to run all the other 4 instances. >> >> and the instances for b.p.o, the meta >> tracker, the Jython tracker, > > > They might not stick with their issue tracker (the Jython team knows what's > going on and they are going to have a discussion over what they will end up > doing). > I (and RDM/MvL) usually apply security patches and other minor fixes to the Jython tracker as well, but otherwise no one maintains it. Since it requires basically no maintenance, I wouldn't mind converting and maintaining this repo together with the other tracker instances, like I already did when we moved from SVN to HG. >> >> and the setuptools tracker > > > Is that being used? https://bitbucket.org/pypa/setuptools/issues seems to be > actively used. > I think this exists to keep record of all issues, but it's not actively used (the bitbucket one is used instead). They probably didn't migrate all the issues to bitbucket, so shutting it down would mean losing all the old issues. As for the Jython tracker, migrating and maintaining this repo with the others shouldn't be a problem. >> >> ) and migrating >> it now means that: > > >> >> 1) The total number of repos to be converted will be 12* instead of 5; >> 2) I (and RDM/MvL) will have to learn a new workflow while updating >> the tracker for the migration; >> >> * 10 if we exclude Rietveld and gae2django (but that means holding up >> the migration until they are removed) >> >> I think it would be best to: >> 1) migrate the other repos; >> 2) work on the b.p.o/github integration using HG; >> 3) remove Rietveld; >> 4) (possibly) migrate the 4 tracker instances to GitHub > > > That's fine by me. As I said, migration is dependent on what happens to > hg.python.org as to whether it has to happen or not. > >> >> >> Our Roundup fork is also an HG clone of upstream Roundup (with a >> separate branch with our changes), so migrating that to git, while >> doable, might be problematic. >> FWIW when we switched from SVN to HG it took over a year and half >> before I migrated the tracker repos, and I don't remember anyone being >> particularly concerned that we were still using SVN or relieved when >> we eventually moved it to HG. >> Personally I wouldn't mind if we kept them on HG (RDM/MvL might >> disagree though), but if it's a burden for the infra-team to keep >> hg.python.org alive, I would prefer if we moved them later rather than >> sooner. > > > Another option is to move it to Bitbucket if you want to keep it on > Mercurial if hg.python.org doesn't stick around. > True. I already have a semi-official clone of it on BitBucket at https://bitbucket.org/ezio_melotti/roundup-bpo/overview >> >> >> > 4. peps [#peps-repo]_ >> > 5. devguide [#devguide-repo]_ >> > 6. cpython [#cpython-repo]_ >> > >> > The devinabox, benchmarksm and tracker repositories are code-only. >> >> Typo: benchmarks >> >> > The peps and devguide repositories involve the generation of webpages. >> > And the cpython repository has special requirements for integration >> > with bugs.python.org [#b.p.o]_. >> > >> > >> > Adding GitHub username support to bugs.python.org >> > +++++++++++++++++++++++++++++++++++++++++++++++++ >> > To keep tracking of CLA signing under the direct control of the PSF, >> > tracking who has signed the PSF CLA will be continued by marking that >> > fact as part of someone's bugs.python.org user profile. What this >> > means is that an association will be needed between a person's >> > bugs.python.org [#b.p.o]_ account and their GitHub account, which >> > will be done through a new field in a user's profile. >> > >> >> We have to decide how to deal with users that don't have a b.p.o account. >> The two options that we discussed in the previous mails are: >> 1) require them to create a b.p.o account; >> 2) allow them to log in to b.p.o using their github account; >> (see also next section) > > > Both require creating an account, it just varies whether they log in using > GitHub or not. I don't see how we can avoid that if we are going to continue > to own the CLA dataset. Honestly I don't see this as a big issue as we have > not seemed to have any issues with people creating accounts up to this > point. > It's not a big issue, but there will be people that only have a github account, and we will need to explain them that in order to accept their PRs we need them to sign the CLA, and in order to sign the CLA we need them to create a b.p.o account and link it to their github username. >> >> > Linking a pull request to an issue >> > ++++++++++++++++++++++++++++++++++ >> > An association between a pull request and an issue is needed to track >> > when a fix has been proposed. The association needs to be many-to-one >> > as there can take multiple pull requests to solve a single issue >> > (technically it should be a many-to-many association for when a >> > single fix solves multiple issues, but this is fairly rare and issues >> > can be merged into one using the ``Superceder`` field on the issue >> > tracker). >> > >> > Association between a pull request and an issue will be done based on >> > detecting the regular expression``[Ii]ssue #(?P\d+)``. If >> > this is specified in either the title or in the body of a message on >> > a pull request then connection will be made on >> > bugs.python.org [#b.p.o]_. A label will also be added to the pull >> > request to signify that the connection was made successfully. This >> > could lead to incorrect associations if the wrong issue or >> > referencing another issue was done, but these are rare occasions. >> > >> >> Is there a way to associate the PR to an issue (in case the user >> forgot) or change association (in case it got the wrong number) after >> the creation of the PR? > > > You tell me. :) I assume any bot we write to handle this will monitor > PR-level comments since GitHub doesn't notify on title changes. But we can > choose any workflow we want, so if we want it to be an explicit command like > `/bot issue 12345` then we can do that instead. > I was asking about the github side. On b.p.o it's not a problem adding PRs either automatically or manually, but I don't know if the same can be done from github (it already happens when a commit message includes the wrong issue number -- the commit can not be changed and the notification is sent to the wrong issue on b.p.o). Even if it's not possible, I guess we could still add a comment to github with the correct issue number, and add the PR to the issue and the people to the nosy list manually. >> >> >> > Create https://git.python.org >> > ''''''''''''''''''''''''''''' >> > Just as hg.python.org [#h.p.o]_ currently points to the Mercurial >> > repository for Python, git.python.org should do the equivalent for >> > the Git repository. >> > >> >> Is this simply a redirect to GitHub or something else? >> We might also want to encourage the use of git.python.org in the >> Devguide and elsewhere to make future migrations simpler. > > > From my understanding it can't be any else but a redirect. But yes, we will > encourage using git.python.org to keep the coupling to GitHub loose for when > we make this kind of change again in the future. > >> >> >> > Backup of pull request data >> > ''''''''''''''''''''''''''' >> > Since GitHub [#github]_ is going to be used for code hosting and code >> > review, those two things need to be backed up. In the case of code >> > hosting, the backup is implicit as all non-shallow Git [#git]_ clones >> > contain the full history of the repository, hence there will be many >> > backups of the repository. >> > >> >> If possible I would still prefer an "official" backup. I don't think >> we want to go around asking who has the latest copy of the repo in >> case something happens to GitHub. > > > I would be shocked if a core developer doesn't have an up-to-date repository > (then again I don't expect to have to flee GitHub overnight, giving us time > to update). But if you want to make it this an optional feature then I'm > fine with that (I don't think the PSF infrastructure team will have issue > with a regularly updated clone of the repository). > Yes, this is not a particularly realistic issue -- even without an official backup we won't lose any code. It's mostly to have an official backup and procedure to restore the repo instead of having to figure out what to do when it happens (even if it probably never will). The infra team can decide if this is a reasonable request or if it's not worth the extra effort. >> >> >> > The code review history does not have the same implicit backup >> > mechanism as the repository itself. That means a daily backup of code >> > review history should be done so that it is not lost in case of any >> > issues with GitHub. It also helps guarantee that a migration from >> > GitHub to some other code review system is feasible were GitHub to >> > disappear overnight. >> > >> >> Is there an API to export this or do we need to write another bot and >> decide how to collect/store the reviews? > > > I don't know; maybe someone else can speak up on this topic who has > experience. I know for a fact that the webhooks can be used, but whether > there is some specific API to get a dump I don't know. > >> >> >> > Test coverage report >> > '''''''''''''''''''' >> > Getting an up-to-date test coverage report for Python's standard >> > library would be extremely beneficial as generating such a report can >> > take quite a while to produce. >> > >> > There are a couple pre-existing services that provide free test >> > coverage for open source projects. Which option is best is an open >> > issue: `Choosing a test coverage service`_. >> > >> >> Do we want to eventually request that all new code introduced is fully >> covered by tests? > > > I have always told sprinters that 80% is a good guideline. I wouldn't ever > want a rule, though. Obviously lowering coverage would not be great and > should be avoided, but if > Maybe we could use red/yellow/green labels to indicate the coverage level. That alone might lead contributors to aim for the green label without having to create and enforce any rule. >> >> I think having an indication of how much code in a PR is covered by >> tests would be useful regardless of the answer to the previous >> question. > > > I know Coveralls already does this; don't know about Codecov. > >> >> >> > >> > Link web content back to files that it is generated from >> > '''''''''''''''''''''''''''''''''''''''''''''''''''''''' >> > It would be helpful for people who find issues with any of the >> > documentation that is generated from a file to have a link on each >> > page which points back to the file on GitHub [#github]_ that stores >> > the content of the page. That would allow for quick pull requests to >> > fix simple things such as spelling mistakes. >> > >> >> Here you are talking about PEPs/devguide/docs.p.o, right? > > > Yes. > FWIW the docs.python.org pages already have a "report a bug" link in the sidebar and also in the footer, but they both just redirect to https://docs.python.org/3/bugs.html . >> >> FWIW this was problem was supposed to be fixed with pootle, but that >> project seems dead (not sure if due to technical reasons, or simply >> because no one had time to integrate it). >> >> > Splitting out parts of the documentation into their own repositories >> > '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' >> > While certain parts of the documentation at https://docs.python.org >> > change with the code, other parts are fairly static and are not >> > tightly bound to the CPython code itself. The following sections of >> > the documentation fit this category of slow-changing, >> > loosely-coupled: >> > >> > * `Tutorial `__ >> > * `Python Setup and Usage >> > `__ >> > * `HOWTOs `__ >> > * `Installing Python Modules >> > `__ >> > * `Distributing Python Modules >> > `__ >> > * `Extending and Embedding >> > `__ >> > * `FAQs `__ >> > >> > These parts of the documentation could be broken out into their own >> > repositories to simplify their maintenance and to expand who has >> > commit rights to them to ease in their maintenance. >> >> I would still consider these somewhat dynamic (especially between Py 2 >> and Py 3). >> There are other documents that are more version-independent, such as >> the whatsnew pages, > > > The What's New docs get updated with changes so that can't be pulled out > from the cpython repo (especially if my dream ever comes true of making > people be better about making sure that document gets updated with changes > that warrant being mentioned there). > The reason to keep them in a separate repo is that they are identical copies of the same document. If you fix a typo in the whatsnew/2.7, you have to fix the same typo in all branches, and in theory the page should always be identical in each branch. Howtos, FAQs, etc. might differ between major and even minor versions as new features are added and old features deprecated, so it makes sense to have (slightly) different versions in each branch (unless you want to move them outside the cpython repo and still keep separate branches). For example https://docs.python.org/3/faq/programming.html#is-there-an-equivalent-of-c-s-ternary-operator had a different answer before 2.5 :) Also moving them might make things more complicated (from simply having to find/clone another repo to building docs for docs.python.org from different sources). >> >> the meta information section at the bottom of >> docs.p.o/index (that includes pages such as >> https://docs.python.org/3/bugs.html ) and a new page with the current >> status of the releases proposed in http://bugs.python.org/issue25296 >> >> > Open Issues >> > =========== >> > For this PEP, open issues are ones where a decision needs to be made >> > to how to approach or solve a problem. Open issues do not entail >> > coordination issues such as who is going to write a certain bit of >> > code. >> > >> > The fate of hg.python.org >> > ------------------------- >> > With the code repositories moving over to Git [#git]_, there is no >> > technical need to keep hg.python.org [#h.p.o]_ running. Having said >> > that, some in the community would like to have it stay functioning as >> > a Mercurial [#hg]_ mirror of the Git repositories. Others have said >> > that they still want a mirror, but one using Git. >> > >> > As maintaining hg.python.org is not necessary, it will be up to the >> > PSF infrastructure committee to decide if they want to spend the >> > time and resources to keep it running. They may also choose whether >> > they want to host a Git mirror on PSF infrastructure. >> > >> >> +1 to keep a read-only mirror. >> Also see above about the tracker repo and our Roundup fork. >> >> > >> > Git CLI commands for committing a pull request to cpython >> > --------------------------------------------------------- >> > Because Git [#git]_ may be a new version control system for core >> > developers, the commands people are expected to run will need to be >> > written down. These commands also need to keep a linear history while >> > giving proper attribution to the pull request author. >> > >> > Another set of commands will also be necessary for when working with >> > a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear >> > history will be kept implicitly, but it will need to make sure to >> > keep/add attribution. >> > >> >> Nick Coghlan, Pierre-Yves David (a Mercurial dev), and Shiyao Ma (one >> of our GSoC student) have been working on an HG extension that >> simplifies interaction with the bug tracker (see the list of patches, >> download/apply them, upload new patches): >> https://bitbucket.org/introom/hg-cpydev >> In a previous email, someone mentioned an alias that allows an easier >> interaction with PRs. Would it make sense to write and distribute an >> official git extension that provides extra commands/aliases for these >> set of commands? (I guess the answer depends on how many tasks we >> have and how straightforward it is to do with plain git commands.) > > > It quite possibly might be. Otherwise shell commands could be written and > kept in Tools/. I major perk IMO with Git over Mercurial is Git Bash comes > with GIt and that gives you Bash on Windows. That makes writing > cross-platform shell scripts to help with this sort of thing easy without > leaving Windows users stranded. > Isn't that also possible with HG, since extensions are written in Python? (BTW, is it possible/reasonable to write git extensions/hooks in Python?) >> >> >> > How to handle the Misc/NEWS file >> > -------------------------------- >> > There are two competing approaches to handling >> > ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on >> > bugs.python.org [#b.p.o]_. This would mean an issue that is marked >> > as "resolved" could not be closed until a news entry is added in the >> > "news" field in the issue tracker. The benefit of tying the news >> > entry to the issue is it makes sure that all changes worthy of a news >> > entry have an accompanying issue. It also makes classifying a news >> > entry automatic thanks to the Component field of the issue. The >> > Versions field of the issue also ties the news entry to which Python >> > releases were affected. A script would be written to query >> > bugs.python.org for relevant new entries for a release and to produce >> > the output needed to be checked into the code repository. This >> > approach is agnostic to whether a commit was done by CLI or bot. >> > >> > The competing approach is to use an individual file per news entry, >> > containg the text for the entry. In this scenario each feature >> >> Typo: containing >> >> > release would have its own directory for news entries and a separate >> > file would be created in that directory that was either named after >> > the issue it closed or a timestamp value (which prevents collisions). >> > Merges across branches would have no issue as the news entry file >> > would still be uniqeuely named and in the directory of the latest >> >> Typo: uniquely >> >> > version that contained the fix. A script would collect all news entry >> > files no matter what directory they reside in and create an >> > appropriate news file (the release directory can be ignored as the >> > mere fact that the file exists is enough to represent that the entry >> > belongs to the release). Classification can either be done by keyword >> > in the new entry file itself or by using subdirectories representing >> > each news entry classification in each release directory (or >> > classification of news entries could be dropped since critical >> > information is captured by the "What's New" documents which are >> > organized). The benefit of this approach is that it keeps the changes >> > with the code that was actually changed. It also ties the message to >> > being part of the commit which introduced the change. For a commit >> > made through the CLI, a script will be provided to help generate the >> > file. In a bot-driven scenario, the merge bot will have a way to >> > specify a specific news entry and create the file as part of its >> > flattened commit (while most likely also supporting using the first >> > line of the commit message if no specific news entry was specified). >> > Code for this approach has been written previously for the Mercurial >> > workflow at http://bugs.python.org/issue18967. There is also a tool >> > from the community at https://pypi.python.org/pypi/towncrier. >> > >> >> Does using git (and fast-forward merges, rebases, etc.) still create >> conflicts? >> (I guess the answer is yes, but perhaps we should double-check.) > > > I believe so, but I guess I could be wrong since I have not explicitly > tested it. > >> >> >> If it does, there's also a third option: writing a merge script. >> I wrote a basic one for hg and it seemed to work decently, perhaps >> with git it's even easier. > > > I'll mention it, but i suspect the file-based solution will win out based on > the feeling I have gotten from people when this topic has come up before. > I was re-reading the issue, and found due interesting links: 1) https://mail.python.org/pipermail/python-dev/2014-December/137393.html (Pierre-Yves David actually convinced me to write the merge script and helped me) 2) https://github.com/twisted/newsbuilder (this can be another option that can be added to the PEP) A few additional points that you might want to add to the PEP: * A new workflow for releasing Python should also be defined, and PEP 101 updated accordingly. * The devguide also needs to be updated. * We should decide what to do with all the other repos at hg.python.org Best Regards, Ezio Melotti >> >> >> > Naming the commit bot >> > --------------------- >> > As naming things can lead to bikeshedding of epic proportions, Brett >> > Cannon will choose the final name of the commit bot (the name of the >> > project for the bot itself can be anything, this is purely for the >> > name used in giving commands to the bot). The name will come from >> > Monty Python, which is only fitting since Python is named after the >> > comedy troupe. It will most likely come from >> > 'Monty Python and the Holy Grail' [#holy-grail]_ (which happens to be >> > how Brett was introduced to Monty Python). Current ideas on the name >> > include: >> > >> > "Black Knight" sketch [#black-knight-sketch]_: >> > >> > * black-knight >> > * none-shall-pass >> > * just-a-flesh-wound >> > >> > "Bridge of Death" sketch [#bridge-of-death-sketch]_: >> > >> > * bridge-keeper >> > * man-from-scene-24 >> > * five-questions >> > * what-is-your-quest >> > * blue-no-green >> > * air-speed-velocity >> > * your-favourite-colour >> > (and that specific spelling; Monty Python is British, after all) >> > >> > "Killer rabbit" sketch [#killer-rabbit-sketch]_: >> > >> > * killer-rabbit >> > * holy-hand-grenade >> > * 5-is-right-out >> > >> > "Witch Village" sketch [#witch-village-sketch]_: >> > >> > * made-of-wood >> > * burn-her >> > >> > "French Taunter" sketch [#french-taunter-sketch]_: >> > >> > * elderberries >> > * kanigget >> > >> > "Constitutional Peasants" sketch [#constitutional-peasants-sketch]_: >> > >> > * dennis >> > * from-the-masses >> > * watery-tart >> > >> > "Knights Who Say Ni" sketch [#ni-sketch]_: >> > >> > * shubbery >> > * ni >> > >> > From "Monty Python and the Holy Grail" in general: >> > >> > * brave-sir-robin >> > >> >> FWIW the bot that currently reports HG commits on IRC is called >> deadparrot. > > > I'll take that into consideration. :) > > -Brett > >> >> >> Best Regards, >> Ezio Melotti >> From brett at python.org Sun Jan 17 21:33:43 2016 From: brett at python.org (Brett Cannon) Date: Mon, 18 Jan 2016 02:33:43 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On Sun, 17 Jan 2016 at 16:30 Ezio Melotti wrote: > On Sun, Jan 17, 2016 at 10:48 PM, Brett Cannon wrote: > > Here is the first complete draft of the PEP laying out what needs to > happen > > to migrate to GitHub (along with some extras that would be nice). The PEP > > can also be viewed at > > > https://github.com/brettcannon/github-transition-pep/blob/master/pep-0512.rst > > or at https://www.python.org/dev/peps/ once the PEP index is updated. > > > > My hope is that I captured all of the requirements for the various > > repositories. If we can agree that is the case then we can start work on > the > > various requirements and start the migration work! > > > > > > ---------- > > PEP: 512 > > Title: Migrating from hg.python.org to GitHub > > Version: $Revision$ > > Last-Modified: $Date$ > > Author: Brett Cannon > > Status: Active > > Type: Process > > Content-Type: text/x-rst > > Created: > > Post-History: 17-Jan-2015 > > > > Abstract > > ======== > > This PEP outlines the steps required to migrate Python's development > > process from Mercurial [#hg]_ as hosted at > > hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting > > the minimum goals of this PEP should allow for the development > > process of Python to be as productive as it currently is, and meeting > > its extended goals should improve it. > > > > Rationale > > ========= > > In 2014, it became obvious that Python's custom development > > process was becoming a hindrance. As an example, for an external > > contributor to submit a fix for a bug that eventually was committed, > > the basic steps were: > > > > 1. Open an issue for the bug at bugs.python.org [#b.p.o]_. > > 2. Checkout out the CPython source code from hg.python.org [#h.p.o]_. > > 3. Make the fix. > > 4. Upload a patch. > > 5. Have a core developer review the patch using our fork of the > > Rietveld code review tool [#rietveld]_. > > 6. Download the patch to make sure it still applies cleanly. > > 7. Run the test suite manually. > > 8. Commit the change manually. > > 9. If the change was for a bugfix release, merge into the > > in-development branch. > > 10. Run the test suite manually again. > > 11. Commit the merge. > > 12. Push the changes. > > > > This is a very heavy, manual process for core developers. Even in the > > simple case, you could only possibly skip the code review step, as you > > would still need to build the documentation. This led to patches > > languishing on the issue tracker due to core developers not being > > able to work through the backlog fast enough to keep up with > > submissions. In turn, that led to a side-effect issue of discouraging > > outside contribution due to frustration from lack of attention, which > > is dangerous problem for an open source project as it runs counter to > > having a viable future for the project. Simply accepting patches > > uploaded to bugs.python.org [#b.p.o]_ is potentially simple for an > > external contributor, it is as slow and burdensome as it gets for > > a core developer to work with. > > > > Hence the decision was made in late 2014 that a move to a new > > development process was needed. A request for PEPs > > proposing new workflows was made, in the end leading to two: > > PEP 481 and PEP 507 proposing GitHub [#github]_ and > > GitLab [#gitlab]_, respectively. > > > > The year 2015 was spent off-and-on working on those proposals and > > trying to tease out details of what made them different from each > > other on the core-workflow mailing list [#core-workflow]_. > > PyCon US 2015 also showed that the community was a bit frustrated > > with our process due to both cognitive overhead for new contributors > > and how long it was taking for core developers to > > look at a patch (see the end of Guido van Rossum's > > keynote at PyCon US 2015 [#guido-keynote]_ as an example of the > > frustration). > > > > On January 1, 2016, the decision was made by Brett Cannon to move the > > development process to GitHub. The key reasons for choosing GitHub > > were [#reasons]_: > > > > * Maintaining custom infrastructure has been a burden on volunteers > > (e.g., a custom fork of Rietveld [#rietveld]_ > > that is not being maintained is currently being used). > > * The custom workflow is very time-consuming for core developers > > (not enough automated tooling built to help support it). > > * The custom workflow is a hindrance to external contributors > > (acts as a barrier of entry due to time required to ramp up on > > development process). > > * There is no feature differentiating GitLab from GitHub beyond > > GitLab being open source. > > * Familiarity with GitHub is far higher amongst core developers and > > external contributors than with GitLab. > > * Our BDFL prefers GitHub (who would be the first person to tell > > you that his opinion shouldn't matter, but the person making the > > decision felt it was important that the BDFL feel comfortable with > > the workflow of his own programming language to encourage his > > continued participation). > > > > There's even already an unofficial image to use to represent the > > migration to GitHub [#pythocat]_. > > > > The overarching goal of this migration is to improve the development > > process to the extent that a core developer can go from external > > contribution submission through all the steps leading to committing > > said contribution all from within a browser on a tablet with WiFi. > > All of this will be done in such a way that if an external > > contributor chooses not to use GitHub then they will continue to have > > that option. > > > > Repositories to Migrate > > ======================= > > While hg.python.org [#h.p.o]_ hosts many repositories, there are only > > six key repositories that must move: > > > > 1. devinabox [#devinabox-repo]_ > > 2. benchmarks [#benchmarks-repo]_ > > 3. tracker [#tracker-repo]_ > > I don't think the tracker repo /must/ move (at least not immediately). > I don't think so either, just that it should probably happen at some point, especially if hg.python.org gets shut down. > The tracker repo includes 7 subrepos (our own forks of Roundup, > Rietveld, and django-gae2django, Is that being used by anyone? > and the instances for b.p.o, the meta > tracker, the Jython tracker, They might not stick with their issue tracker (the Jython team knows what's going on and they are going to have a discussion over what they will end up doing). > and the setuptools tracker Is that being used? https://bitbucket.org/pypa/setuptools/issues seems to be actively used. > ) and migrating > it now means that: > 1) The total number of repos to be converted will be 12* instead of 5; > 2) I (and RDM/MvL) will have to learn a new workflow while updating > the tracker for the migration; > > * 10 if we exclude Rietveld and gae2django (but that means holding up > the migration until they are removed) > > I think it would be best to: > 1) migrate the other repos; > 2) work on the b.p.o/github integration using HG; > 3) remove Rietveld; > 4) (possibly) migrate the 4 tracker instances to GitHub > That's fine by me. As I said, migration is dependent on what happens to hg.python.org as to whether it has to happen or not. > > Our Roundup fork is also an HG clone of upstream Roundup (with a > separate branch with our changes), so migrating that to git, while > doable, might be problematic. > FWIW when we switched from SVN to HG it took over a year and half > before I migrated the tracker repos, and I don't remember anyone being > particularly concerned that we were still using SVN or relieved when > we eventually moved it to HG. > Personally I wouldn't mind if we kept them on HG (RDM/MvL might > disagree though), but if it's a burden for the infra-team to keep > hg.python.org alive, I would prefer if we moved them later rather than > sooner. > Another option is to move it to Bitbucket if you want to keep it on Mercurial if hg.python.org doesn't stick around. > > > 4. peps [#peps-repo]_ > > 5. devguide [#devguide-repo]_ > > 6. cpython [#cpython-repo]_ > > > > The devinabox, benchmarksm and tracker repositories are code-only. > > Typo: benchmarks > > > The peps and devguide repositories involve the generation of webpages. > > And the cpython repository has special requirements for integration > > with bugs.python.org [#b.p.o]_. > > > > Migration Plan > > ============== > > The migration plan is separated into sections based on what is > > required to migrate the repositories listed in the > > `Repositories to Migrate`_ section. Completion of requirements > > outlined in each section should unblock the migration of the related > > repositories. The sections are expected to be completed in order, but > > not necessarily the requirements within a section. > > > > Requirements for Code-Only Repositories > > --------------------------------------- > > Completion of the requirements in this section will allow the > > devinabox, benchmarks, and tracker repositories to move to > > GitHub. While devinabox has a sufficiently descriptive name, the > > benchmarks repository does not; therefore, it will be named > > "python-benchmark-suite". The tracker repo also has a misleading name > > and will be renamed "bugs.python.org". > > As mentioned above, tracker contains 7 subrepos, and the > bugs.python.org instance is tracker/python-dev. > Roundup (tracker/roundup) is running 4 tracker instances > (tracker/python-dev, tracker/meta for the metatracker, tracker/jython > for the Jython tracker, and tracker/setuptools for > https://bugs.python.org/setuptools/ (now hosted on BitBucket but kept > around for historical reasons)). > > > > > Create a 'python-dev' team > > '''''''''''''''''''''''''' > > To manage permissions, a 'python-dev' team will be created as part of > > the python organization [#github-python-org]_. Any repository that is > > moved will have the 'python-dev' team added to it with write > > permissions [#github-org-perms]_. Anyone who previously had rights to > > manage SSH keys on hg.python.org will become a team maintainer for the > > 'python-dev' team. > > > > Define commands to move a Mercurial repository to Git > > ''''''''''''''''''''''''''''''''''''''''''''''''''''' > > Since moving to GitHub also entails moving to Git [#git]_, we must > > decide what tools and commands we will run to translate a Mercurial > > repository to Git. The exact tools and steps to use are an > > open issue; see `Tools and commands to move from Mercurial to Git`_. > > > > CLA enforcement > > ''''''''''''''' > > A key part of any open source project is making sure that its source > > code can be properly licensed. This requires making sure all people > > making contributions have signed a contributor license agreement > > (CLA) [#cla]_. Up until now, enforcement of CLA signing of > > contributed code has been enforced by core developers checking > > whether someone had an ``*`` by their username on > > bugs.python.org [#b.p.o]_. With this migration, the plan is to start > > off with automated checking and enforcement of contributors signing > > the CLA. > > > > Adding GitHub username support to bugs.python.org > > +++++++++++++++++++++++++++++++++++++++++++++++++ > > To keep tracking of CLA signing under the direct control of the PSF, > > tracking who has signed the PSF CLA will be continued by marking that > > fact as part of someone's bugs.python.org user profile. What this > > means is that an association will be needed between a person's > > bugs.python.org [#b.p.o]_ account and their GitHub account, which > > will be done through a new field in a user's profile. > > > > We have to decide how to deal with users that don't have a b.p.o account. > The two options that we discussed in the previous mails are: > 1) require them to create a b.p.o account; > 2) allow them to log in to b.p.o using their github account; > (see also next section) > Both require creating an account, it just varies whether they log in using GitHub or not. I don't see how we can avoid that if we are going to continue to own the CLA dataset. Honestly I don't see this as a big issue as we have not seemed to have any issues with people creating accounts up to this point. > > > A bot to enforce CLA signing > > ++++++++++++++++++++++++++++ > > With an association between someone's GitHub account and their > > bugs.python.org [#b.p.o]_ account, which has the data as to whether > > someone has signed the CLA, a bot can monitor pull requests on > > GitHub and denote whether the contributor has signed the CLA. > > > > If the user has signed the CLA, the bot will add a positive label to > > the issue to denote the pull request has no CLA issues (e.g., a green > > label stating, "CLA: ?"). If the contributor has not signed a CLA, > > a negative label will be added to the pull request will be blocked > > using GitHub's status API (e.g., a red label stating, "CLA: ?"). If a > > contributor lacks a bugs.python.org account, that will lead to > > another label (e.g., "CLA: ? (no account)"). Using a label for both > > positive and negative cases provides a fallback notification if the > > bot happens to fail, preventing potential false-positives or > > false-negatives. It also allows for an easy way to trigger the bot > > again by simply removing a CLA-related label. > > > > Requirements for Web-Related Repositories > > ----------------------------------------- > > Due to their use for generating webpages, the > > devguide [#devguide-repo]_ and peps [#peps-repo]_ repositories need > > their respective processes updated to pull from their new Git > > repositories. > > > > The devguide repository might also need to be named > > ``python-devguide`` to make sure the repository is not ambiguous > > when viewed in isolation from the > > python organization [#github-python-org]_. > > > > Requirements for the cpython Repository > > --------------------------------------- > > Obviously the most active and important repository currently hosted > > at hg.python.org [#h.p.o]_ is the cpython > > repository [#cpython-repo]_. Because of its importance and high- > > frequency use, it requires more tooling before being moved to GitHub > > compared to the other repositories mentioned in this PEP. > > > > Document steps to commit a pull request > > ''''''''''''''''''''''''''''''''''''''' > > During the process of choosing a new development workflow, it was > > decided that a linear history is desired. This means that the > > convenient "Merge" button in GitHub pull requests is undesireable, as > > Typo: undesirable > > > it creates a merge commit along with all of the contributor's > > individual commits (this does not affect the other repositories where > > the desire for a linear history doesn't exist). > > > > Luckily, Git [#git]_ does not require GitHub's workflow and so one can > > be chosen which gives us a linear history by using Git's CLI. The > > expectation is that all pull requests will be fast-forwarded and > > rebased before being pushed to the master repository. This should > > give proper attribution to the pull request author in the Git > > history. > > > > A second set of recommended commands will also be written for > > committing a contribution from a patch file uploaded to > > bugs.python.org [#b.p.o]_. This will obviously help keep the linear > > history, but it will need to be made to have attribution to the patch > > author. > > > > The exact sequence of commands that will be given as guidelines to > > core developers is an open issue: > > `Git CLI commands for committing a pull request to cpython`_. > > > > Handling Misc/NEWS > > '''''''''''''''''' > > Traditionally the ``Misc/NEWS`` file [#news-file]_ has been problematic > > for changes which spanned Python releases. Often times there will be > > merge conflicts when committing a change between e.g., 3.5 and 3.6 > > only in the ``Misc/NEWS`` file. It's so common, in fact, that the > > example instructions in the devguide explicitly mention how to > > resolve conflicts in the ``Misc/NEWS`` file > > [#devguide-merge-across-branches]_. As part of our tool > > modernization, working with the ``Misc/NEWS`` file will be > > simplified. > > > > There are currently two competing approaches to solving the > > ``Misc/NEWS`` problem which are discussed in an open issue: > > `How to handle the Misc/NEWS file`_. > > > > Handling Misc/ACKS > > '''''''''''''''''' > > Traditionally the ``Misc/ACKS`` file [#acks-file]_ has been managed > > by hand. But thanks to Git supporting an ``author`` value as well as > > a ``committer`` value per commit, authorship of a commit can be part > > of the history of the code itself. > > > > As such, manual management of ``Misc/ACKS`` will become optional. A > > script will be written that will collect all author and committer > > names and merge them into ``Misc/ACKS`` with all of the names listed > > prior to the move to Git. Running this script will become part of the > > release process. > > > > Linking pull requests to issues > > ''''''''''''''''''''''''''''''' > > Historically, external contributions were attached to an issue on > > bugs.python.org [#b.p.o]_ thanks to the fact that all external > > contributions were uploaded as a file. For changes committed by a > > core developer who committed a change directly, the specifying of an > > issue number in the commit message of the format ``Issue #`` at the > > start of the message led to a comment being posted to the issue > > linking to the commit. > > > > Linking a pull request to an issue > > ++++++++++++++++++++++++++++++++++ > > An association between a pull request and an issue is needed to track > > when a fix has been proposed. The association needs to be many-to-one > > as there can take multiple pull requests to solve a single issue > > (technically it should be a many-to-many association for when a > > single fix solves multiple issues, but this is fairly rare and issues > > can be merged into one using the ``Superceder`` field on the issue > > tracker). > > > > Association between a pull request and an issue will be done based on > > detecting the regular expression``[Ii]ssue #(?P\d+)``. If > > this is specified in either the title or in the body of a message on > > a pull request then connection will be made on > > bugs.python.org [#b.p.o]_. A label will also be added to the pull > > request to signify that the connection was made successfully. This > > could lead to incorrect associations if the wrong issue or > > referencing another issue was done, but these are rare occasions. > > > > Is there a way to associate the PR to an issue (in case the user > forgot) or change association (in case it got the wrong number) after > the creation of the PR? > You tell me. :) I assume any bot we write to handle this will monitor PR-level comments since GitHub doesn't notify on title changes. But we can choose any workflow we want, so if we want it to be an explicit command like `/bot issue 12345` then we can do that instead. > > > Notify the issue if the pull request is committed > > +++++++++++++++++++++++++++++++++++++++++++++++++ > > Once a pull request is closed (merged or not), the issue should be > > updated to reflect this fact. > > > > Update linking service for mapping commit IDs to URLs > > ''''''''''''''''''''''''''''''''''''''''''''''''''''' > > Currently you can use https://hg.python.org/lookup/ with a revision > > ID from either the Subversion or Mercurial copies of the > > cpython repo [#cpython-repo]_ to get redirected to the URL for that > > revision in the Mercurial repository. The URL rewriter will need to > > be updated to redirect to the Git repository and to support the new > > revision IDs created for the Git repository. > > > > Create https://git.python.org > > ''''''''''''''''''''''''''''' > > Just as hg.python.org [#h.p.o]_ currently points to the Mercurial > > repository for Python, git.python.org should do the equivalent for > > the Git repository. > > > > Is this simply a redirect to GitHub or something else? > We might also want to encourage the use of git.python.org in the > Devguide and elsewhere to make future migrations simpler. > >From my understanding it can't be any else but a redirect. But yes, we will encourage using git.python.org to keep the coupling to GitHub loose for when we make this kind of change again in the future. > > > Backup of pull request data > > ''''''''''''''''''''''''''' > > Since GitHub [#github]_ is going to be used for code hosting and code > > review, those two things need to be backed up. In the case of code > > hosting, the backup is implicit as all non-shallow Git [#git]_ clones > > contain the full history of the repository, hence there will be many > > backups of the repository. > > > > If possible I would still prefer an "official" backup. I don't think > we want to go around asking who has the latest copy of the repo in > case something happens to GitHub. > I would be shocked if a core developer doesn't have an up-to-date repository (then again I don't expect to have to flee GitHub overnight, giving us time to update). But if you want to make it this an optional feature then I'm fine with that (I don't think the PSF infrastructure team will have issue with a regularly updated clone of the repository). > > > The code review history does not have the same implicit backup > > mechanism as the repository itself. That means a daily backup of code > > review history should be done so that it is not lost in case of any > > issues with GitHub. It also helps guarantee that a migration from > > GitHub to some other code review system is feasible were GitHub to > > disappear overnight. > > > > Is there an API to export this or do we need to write another bot and > decide how to collect/store the reviews? > I don't know; maybe someone else can speak up on this topic who has experience. I know for a fact that the webhooks can be used, but whether there is some specific API to get a dump I don't know. > > > Change sys._mercurial > > ''''''''''''''''''''' > > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` > > attribute will need to be removed. An equivalent ``sys._git`` > > attribute will be needed to take its place. > > > > Optional, Planned Features > > -------------------------- > > Once the cpython repository [#cpython-repo]_ is migrated, all > > repositories will have been moved to GitHub [#github]_ and the > > development process should be on equal footing as before. But a key > > reason for this migration is to improve the development process, > > making it better than it has ever been. This section outlines some > > plans on how to improve things. > > > > It should be mentioned that overall feature planning for > > bugs.python.org [#b.p.o]_ -- which includes plans independent of this > > migration -- are tracked on their own wiki page [#tracker-plans]_. > > > > Bot to handle pull request merging > > '''''''''''''''''''''''''''''''''' > > As stated in the section entitled > > "`Document steps to commit a pull request`_", the desire is to > > maintain a linear history for cpython. Unfortunately, > > Github's [#github]_ web-based workflow does not support a linear > > history. Because of this, a bot should be written to substitute for > > GitHub's in-browser commit abilities. > > > > To start, the bot should accept commands to commit a pull request > > against a list of branches. This allows for committing a pull request > > that fixes a bug in multiple versions of Python. > > > > More advanced features such as a commit queue can come later. This > > would linearly apply accepted pull requests and verify that the > > commits did not interfere with each other by running the test suite > > and backing out commits if the test run failed. To help facilitate > > the speed of testing, all patches committed since the last test run > > can be applied and run in a single test run as the optimistic > > assumption is that the patches will work in tandem. > > > > Inspiration or basis of the bot could be taken from pre-existig bots > > such as Homu [#homu]_ or Zuul [#zuul]_. > > > > The name given to this bot in order to give it commands is an open > > issue: `Naming the commit bot`_. > > > > Continuous integration per pull request > > ''''''''''''''''''''''''''''''''''''''' > > To help speed up pull request approvals, continuous integration > > testing should be used. This helps mitigate the need for a core > > developer to download a patch simply to run the test suite against > > the patch. > > > > Which free CI service to use is an open issue: > > `Choosing a CI service`_. > > > > Test coverage report > > '''''''''''''''''''' > > Getting an up-to-date test coverage report for Python's standard > > library would be extremely beneficial as generating such a report can > > take quite a while to produce. > > > > There are a couple pre-existing services that provide free test > > coverage for open source projects. Which option is best is an open > > issue: `Choosing a test coverage service`_. > > > > Do we want to eventually request that all new code introduced is fully > covered by tests? > I have always told sprinters that 80% is a good guideline. I wouldn't ever want a rule, though. Obviously lowering coverage would not be great and should be avoided, but if > I think having an indication of how much code in a PR is covered by > tests would be useful regardless of the answer to the previous > question. > I know Coveralls already does this; don't know about Codecov. > > > Notifying issues of pull request comments > > ''''''''''''''''''''''''''''''''''''''''' > > The current development process does not include notifying an issue > > on bugs.python.org [#b.p.o]_ when a review comment is left on > > Rietveld [#rietveld]_. It would be nice to fix this so that people > > can subscribe only to comments at bugs.python.org and not > > GitHub [#github]_ and yet still know when something occurs on GitHub > > in terms of review comments on relevant pull requests. Current > > thinking is to post a comment to bugs.python.org to the relevant > > issue when at least one review comment has been made over a certain > > period of time (e.g., 15 or 30 minutes). This keeps the email volume > > down for those that receive both GitHub and bugs.python.org email > > notifications while still making sure that those only following > > bugs.python.org know when there might be a review comment to address. > > > > Allow bugs.python.org to use GitHub as a login provider > > ''''''''''''''''''''''''''''''''''''''''''''''''''''''' > > As of right now, bugs.python.org [#b.p.o]_ allows people to log in > > using Google, Launchpad, or OpenID credentials. It would be good to > > expand this to GitHub credentials. > > > > Web hooks for re-generating web content > > ''''''''''''''''''''''''''''''''''''''' > > The content at https://docs.python.org/, > > https://docs.python.org/devguide, and > > https://www.python.org/dev/peps/ are all derived from files kept in > > one of the repositories to be moved as part of this migration. As > > such, it would be nice to set up appropriate webhooks to trigger > > rebuilding the appropriate web content when the files they are based > > on change instead of having to wait for, e.g., a cronjob to trigger. > > > > Link web content back to files that it is generated from > > '''''''''''''''''''''''''''''''''''''''''''''''''''''''' > > It would be helpful for people who find issues with any of the > > documentation that is generated from a file to have a link on each > > page which points back to the file on GitHub [#github]_ that stores > > the content of the page. That would allow for quick pull requests to > > fix simple things such as spelling mistakes. > > > > Here you are talking about PEPs/devguide/docs.p.o, right? > Yes. > FWIW this was problem was supposed to be fixed with pootle, but that > project seems dead (not sure if due to technical reasons, or simply > because no one had time to integrate it). > > > Splitting out parts of the documentation into their own repositories > > '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' > > While certain parts of the documentation at https://docs.python.org > > change with the code, other parts are fairly static and are not > > tightly bound to the CPython code itself. The following sections of > > the documentation fit this category of slow-changing, > > loosely-coupled: > > > > * `Tutorial `__ > > * `Python Setup and Usage >`__ > > * `HOWTOs `__ > > * `Installing Python Modules > > `__ > > * `Distributing Python Modules > > `__ > > * `Extending and Embedding > > `__ > > * `FAQs `__ > > > > These parts of the documentation could be broken out into their own > > repositories to simplify their maintenance and to expand who has > > commit rights to them to ease in their maintenance. > > I would still consider these somewhat dynamic (especially between Py 2 > and Py 3). > There are other documents that are more version-independent, such as > the whatsnew pages, The What's New docs get updated with changes so that can't be pulled out from the cpython repo (especially if my dream ever comes true of making people be better about making sure that document gets updated with changes that warrant being mentioned there). > the meta information section at the bottom of > docs.p.o/index (that includes pages such as > https://docs.python.org/3/bugs.html ) and a new page with the current > status of the releases proposed in http://bugs.python.org/issue25296 > > > > > Status > > ====== > > Requirements for migrating the devinabox [#devinabox-repo]_, > > benchmarks [#benchmarks-repo]_, and tracker [#tracker-repo]_ > > repositories: > > > > * Not started > > > > - `Create a 'python-dev' team`_ > > - `Define commands to move a Mercurial repository to Git`_ > > - `Adding GitHub username support to bugs.python.org`_ > > - `A bot to enforce CLA signing`_ > > > > * In progress > > > > - None > > > > * Completed > > > > - None > > > > Repositories whose build steps need updating: > > > > * Not started > > > > - peps [#peps-repo]_ > > - devguide [#devguide-repo]_ > > > > * In progress > > > > - None > > > > * Completed > > > > - None > > > > Requirements to move over the cpython repo [#cpython-repo]_: > > > > * Not started > > > > - `Document steps to commit a pull request`_ > > - `Handling Misc/NEWS`_ > > - `Handling Misc/ACKS`_ > > - `Linking a pull request to an issue`_ > > - `Notify the issue if the pull request is committed`_ > > - `Update linking service for mapping commit IDs to URLs`_ > > - `Create https://git.python.org`_ > > - `Backup of pull request data`_ > > - `Change sys._mercurial`_ > > > > * In progress > > > > - None > > > > * Completed > > > > - None > > > > Optional features: > > > > * Not started > > > > - `Bot to handle pull request merging`_ > > - `Continuous integration per pull request`_ > > - `Test coverage report`_ > > - `Notifying issues of pull request comments`_ > > - `Allow bugs.python.org to use GitHub as a login provider`_ > > - `Web hooks for re-generating web content`_ > > - `Link web content back to files that it is generated from`_ > > - `Splitting out parts of the documentation into their own > repositories`_ > > > > * In progress > > > > - None > > > > * Completed > > > > - None > > > > Open Issues > > =========== > > For this PEP, open issues are ones where a decision needs to be made > > to how to approach or solve a problem. Open issues do not entail > > coordination issues such as who is going to write a certain bit of > > code. > > > > The fate of hg.python.org > > ------------------------- > > With the code repositories moving over to Git [#git]_, there is no > > technical need to keep hg.python.org [#h.p.o]_ running. Having said > > that, some in the community would like to have it stay functioning as > > a Mercurial [#hg]_ mirror of the Git repositories. Others have said > > that they still want a mirror, but one using Git. > > > > As maintaining hg.python.org is not necessary, it will be up to the > > PSF infrastructure committee to decide if they want to spend the > > time and resources to keep it running. They may also choose whether > > they want to host a Git mirror on PSF infrastructure. > > > > +1 to keep a read-only mirror. > Also see above about the tracker repo and our Roundup fork. > > > Tools and commands to move from Mercurial to Git > > ------------------------------------------------ > > A decision needs to be made on exactly what tooling and what commands > > involving those tools will be used to convert a Mercurial repository > > to Git. Currently a suggestion has been made to use > > https://github.com/frej/fast-export. Another suggestion is to use > > https://github.com/felipec/git-remote-hg. Finally, > > http://hg-git.github.io/ has been suggested. > > > > Git CLI commands for committing a pull request to cpython > > --------------------------------------------------------- > > Because Git [#git]_ may be a new version control system for core > > developers, the commands people are expected to run will need to be > > written down. These commands also need to keep a linear history while > > giving proper attribution to the pull request author. > > > > Another set of commands will also be necessary for when working with > > a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear > > history will be kept implicitly, but it will need to make sure to > > keep/add attribution. > > > > Nick Coghlan, Pierre-Yves David (a Mercurial dev), and Shiyao Ma (one > of our GSoC student) have been working on an HG extension that > simplifies interaction with the bug tracker (see the list of patches, > download/apply them, upload new patches): > https://bitbucket.org/introom/hg-cpydev > In a previous email, someone mentioned an alias that allows an easier > interaction with PRs. Would it make sense to write and distribute an > official git extension that provides extra commands/aliases for these > set of commands? (I guess the answer depends on how many tasks we > have and how straightforward it is to do with plain git commands.) > It quite possibly might be. Otherwise shell commands could be written and kept in Tools/. I major perk IMO with Git over Mercurial is Git Bash comes with GIt and that gives you Bash on Windows. That makes writing cross-platform shell scripts to help with this sort of thing easy without leaving Windows users stranded. > > > How to handle the Misc/NEWS file > > -------------------------------- > > There are two competing approaches to handling > > ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on > > bugs.python.org [#b.p.o]_. This would mean an issue that is marked > > as "resolved" could not be closed until a news entry is added in the > > "news" field in the issue tracker. The benefit of tying the news > > entry to the issue is it makes sure that all changes worthy of a news > > entry have an accompanying issue. It also makes classifying a news > > entry automatic thanks to the Component field of the issue. The > > Versions field of the issue also ties the news entry to which Python > > releases were affected. A script would be written to query > > bugs.python.org for relevant new entries for a release and to produce > > the output needed to be checked into the code repository. This > > approach is agnostic to whether a commit was done by CLI or bot. > > > > The competing approach is to use an individual file per news entry, > > containg the text for the entry. In this scenario each feature > > Typo: containing > > > release would have its own directory for news entries and a separate > > file would be created in that directory that was either named after > > the issue it closed or a timestamp value (which prevents collisions). > > Merges across branches would have no issue as the news entry file > > would still be uniqeuely named and in the directory of the latest > > Typo: uniquely > > > version that contained the fix. A script would collect all news entry > > files no matter what directory they reside in and create an > > appropriate news file (the release directory can be ignored as the > > mere fact that the file exists is enough to represent that the entry > > belongs to the release). Classification can either be done by keyword > > in the new entry file itself or by using subdirectories representing > > each news entry classification in each release directory (or > > classification of news entries could be dropped since critical > > information is captured by the "What's New" documents which are > > organized). The benefit of this approach is that it keeps the changes > > with the code that was actually changed. It also ties the message to > > being part of the commit which introduced the change. For a commit > > made through the CLI, a script will be provided to help generate the > > file. In a bot-driven scenario, the merge bot will have a way to > > specify a specific news entry and create the file as part of its > > flattened commit (while most likely also supporting using the first > > line of the commit message if no specific news entry was specified). > > Code for this approach has been written previously for the Mercurial > > workflow at http://bugs.python.org/issue18967. There is also a tool > > from the community at https://pypi.python.org/pypi/towncrier. > > > > Does using git (and fast-forward merges, rebases, etc.) still create > conflicts? > (I guess the answer is yes, but perhaps we should double-check.) > I believe so, but I guess I could be wrong since I have not explicitly tested it. > > If it does, there's also a third option: writing a merge script. > I wrote a basic one for hg and it seemed to work decently, perhaps > with git it's even easier. > I'll mention it, but i suspect the file-based solution will win out based on the feeling I have gotten from people when this topic has come up before. > > > Naming the commit bot > > --------------------- > > As naming things can lead to bikeshedding of epic proportions, Brett > > Cannon will choose the final name of the commit bot (the name of the > > project for the bot itself can be anything, this is purely for the > > name used in giving commands to the bot). The name will come from > > Monty Python, which is only fitting since Python is named after the > > comedy troupe. It will most likely come from > > 'Monty Python and the Holy Grail' [#holy-grail]_ (which happens to be > > how Brett was introduced to Monty Python). Current ideas on the name > > include: > > > > "Black Knight" sketch [#black-knight-sketch]_: > > > > * black-knight > > * none-shall-pass > > * just-a-flesh-wound > > > > "Bridge of Death" sketch [#bridge-of-death-sketch]_: > > > > * bridge-keeper > > * man-from-scene-24 > > * five-questions > > * what-is-your-quest > > * blue-no-green > > * air-speed-velocity > > * your-favourite-colour > > (and that specific spelling; Monty Python is British, after all) > > > > "Killer rabbit" sketch [#killer-rabbit-sketch]_: > > > > * killer-rabbit > > * holy-hand-grenade > > * 5-is-right-out > > > > "Witch Village" sketch [#witch-village-sketch]_: > > > > * made-of-wood > > * burn-her > > > > "French Taunter" sketch [#french-taunter-sketch]_: > > > > * elderberries > > * kanigget > > > > "Constitutional Peasants" sketch [#constitutional-peasants-sketch]_: > > > > * dennis > > * from-the-masses > > * watery-tart > > > > "Knights Who Say Ni" sketch [#ni-sketch]_: > > > > * shubbery > > * ni > > > > From "Monty Python and the Holy Grail" in general: > > > > * brave-sir-robin > > > > FWIW the bot that currently reports HG commits on IRC is called deadparrot. > I'll take that into consideration. :) -Brett > > Best Regards, > Ezio Melotti > > > Choosing a CI service > > --------------------- > > There are various CI services that provide free support for open > > source projects hosted on GitHub [#github]_. Two such examples are > > Travis [#travis]_ and Codeship [#codeship]_. Whatever solution is > > chosen will need to not time out in the time it takes to execute > > Python's test suite. It should optimally provide access to multiple C > > compilers for more thorough testing. Network access is also > > beneficial. > > > > The current CI service for Python is Pypatcher [#pypatcher]_. A > > request can be made in IRC to try a patch from > > bugs.python.org [#b.p.o]_. The results can be viewed at > > https://ci.centos.org/job/cPython-build-patch/ . > > > > Choosing a test coverage service > > -------------------------------- > > Getting basic test coverage of Python's standard library can be > > created simply by using coverage.py [#coverage]_. Getting > > thorough test coverage is actually quite tricky, with the details > > outlined in the devinabox's README [#devinabox-repo]_. It would be > > best if a service could be found that would allow for thorough test > > coverage, but it might not be feasible. > > > > Free test coverage services include Coveralls [#coveralls]_ and > > Codecov [#codecov]_. > > > > Rejected Ideas > > ============== > > Separate Python 2 and Python 3 repositories > > ------------------------------------------- > > It was discussed whether separate repositories for Python 2 and > > Python 3 were desired. The thinking was that this would shrink the > > overall repository size which benefits people with slow Internet > > connections or small bandwidth caps. > > > > In the end it was decided that it was easier logistically to simply > > keep all of CPython's history in a single repository. > > > > Commit multi-release changes in bugfix branch first > > --------------------------------------------------- > > As the current development process has changes committed in the > > oldest branch first and then merged up to the default branch, the > > question came up as to whether this workflow should be perpetuated. > > In the end it was decided that committing in the newest branch and > > then cherrypicking changes into older branches would work best as > > most people will instinctively work off the newest branch and it is a > > more common workflow when using Git [#git]_. > > > > Deriving ``Misc/NEWS`` from the commit logs > > ------------------------------------------- > > As part of the discussion surrounding `Handling Misc/NEWS`_, the > > suggestion has come up of deriving the file from the commit logs > > itself. In this scenario, the first line of a commit message would be > > taken to represent the news entry for the change. Some heuristic to > > tie in whether a change warranted a news entry would be used, e.g., > > whether an issue number is listed. > > > > This idea has been rejected due to some core developers preferring to > > write a news entry separate from the commit message. The argument is > > the first line of a commit message compared to that of a news entry > > have different requirements in terms of brevity, what should be said, > > etc. > > > > References > > ========== > > .. [#h.p.o] https://hg.python.org > > > > .. [#GitHub] GitHub (https://github.com) > > > > .. [#hg] Mercurial (https://www.mercurial-scm.org/) > > > > .. [#git] Git (https://git-scm.com/) > > > > .. [#b.p.o] https://bugs.python.org > > > > .. [#rietveld] Rietveld (https://github.com/rietveld-codereview/rietveld > ) > > > > .. [#gitlab] GitLab (https://about.gitlab.com/) > > > > .. [#core-workflow] core-workflow mailing list > > (https://mail.python.org/mailman/listinfo/core-workflow) > > > > .. [#guido-keynote] Guido van Rossum's keynote at PyCon US > > (https://www.youtube.com/watch?v=G-uKNd5TSBw) > > > > .. [#reasons] Email to core-workflow outlining reasons why GitHub was > > selected > > > > ( > https://mail.python.org/pipermail/core-workflow/2016-January/000345.html) > > > > .. [#benchmarks-repo] Mercurial repository for the Unified Benchmark > Suite > > (https://hg.python.org/benchmarks/) > > > > .. [#devinabox-repo] Mercurial repository for devinabox > > (https://hg.python.org/devinabox/) > > > > .. [#peps-repo] Mercurial repository of the Python Enhancement Proposals > > (https://hg.python.org/peps/) > > > > .. [#tracker-repo] bugs.python.org code repository > > (https://hg.python.org/tracker/python-dev/) > > > > .. [#devguide-repo] Mercurial repository for the Python Developer's Guide > > (https://hg.python.org/devguide/) > > > > .. [#cpython-repo] Mercurial repository for CPython > > (https://hg.python.org/cpython/) > > > > .. [#github-python-org] Python organization on GitHub > > (https://github.com/python) > > > > .. [#github-org-perms] GitHub repository permission levels > > > > ( > https://help.github.com/enterprise/2.4/user/articles/repository-permission-levels-for-an-organization/ > ) > > > > .. [#cla] Python Software Foundation Contributor Agreement > > (https://www.python.org/psf/contrib/contrib-form/) > > > > .. [#news-file] ``Misc/NEWS`` > > (https://hg.python.org/cpython/file/default/Misc/NEWS) > > > > .. [#acks-file] ``Misc/ACKS`` > > (https://hg.python.org/cpython/file/default/Misc/ACKS) > > > > .. [#devguide-merge-across-branches] Devguide instructions on how to > merge > > across branches > > > > ( > https://docs.python.org/devguide/committing.html#merging-between-different-branches-within-the-same-major-version > ) > > > > .. [#pythocat] Pythocat (https://octodex.github.com/pythocat/) > > > > .. [#tracker-plans] Wiki page for bugs.python.org feature development > > (https://wiki.python.org/moin/TrackerDevelopmentPlanning) > > > > .. [#black-knight-sketch] The "Black Knight" sketch from "Monty Python > and > > the Holy Grail" > > (https://www.youtube.com/watch?v=dhRUe-gz690) > > > > .. [#bridge-of-death-sketch] The "Bridge of Death" sketch from "Monty > Python > > and the Holy Grail" > > (https://www.youtube.com/watch?v=cV0tCphFMr8) > > > > .. [#holy-grail] "Monty Python and the Holy Grail" sketches > > > > ( > https://www.youtube.com/playlist?list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp) > > > > .. [#killer-rabbit-sketch] "Killer rabbit" sketch from "Monty Python and > the > > Holy Grail" > > > > ( > https://www.youtube.com/watch?v=Nvs5pqf-DMA&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=11 > ) > > > > .. [#witch-village-sketch] "Witch Village" from "Monty Python and the > Holy > > Grail" > > > > ( > https://www.youtube.com/watch?v=k3jt5ibfRzw&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=12 > ) > > > > .. [#french-taunter-sketch] "French Taunter" from "Monty Python and the > Holy > > Grail" > > > > ( > https://www.youtube.com/watch?v=A8yjNbcKkNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=13 > ) > > > > .. [#constitutional-peasants-sketch] "Constitutional Peasants" from > "Monty > > Python and the Holy Grail" > > > > ( > https://www.youtube.com/watch?v=JvKIWjnEPNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=14 > ) > > > > .. [#ni-sketch] "Knights Who Say Ni" from "Monty Python and the Holy > Grail" > > > > ( > https://www.youtube.com/watch?v=zIV4poUZAQo&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=15 > ) > > > > .. [#homu] Homu (http://homu.io/) > > > > .. [#zuul] Zuul (http://docs.openstack.org/infra/zuul/) > > > > .. [#travis] Travis (https://travis-ci.org/) > > > > .. [#codeship] Codeship (https://codeship.com/) > > > > .. [#coverage] coverage.py (https://pypi.python.org/pypi/coverage) > > > > .. [#coveralls] Coveralls (https://coveralls.io/) > > > > .. [#codecov] Codecov (https://codecov.io/) > > > > .. [#pypatcher] Pypatcher (https://github.com/kushaldas/pypatcher) > > > > Copyright > > ========= > > > > This document has been placed in the public domain. > > > > > > > > .. > > Local Variables: > > mode: indented-text > > indent-tabs-mode: nil > > sentence-end-double-space: t > > fill-column: 70 > > coding: utf-8 > > End: > > > > > > _______________________________________________ > > core-workflow mailing list > > core-workflow at python.org > > https://mail.python.org/mailman/listinfo/core-workflow > > This list is governed by the PSF Code of Conduct: > > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jan 18 12:26:15 2016 From: brett at python.org (Brett Cannon) Date: Mon, 18 Jan 2016 17:26:15 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: [just an FYI to everyone: replying without trimming the PEP will lead to moderation, so please cut out stuff that doesn't matter when replying] On Sun, 17 Jan 2016 at 21:37 Ezio Melotti wrote: > On Mon, Jan 18, 2016 at 4:33 AM, Brett Cannon wrote: > > > > > > On Sun, 17 Jan 2016 at 16:30 Ezio Melotti > wrote: > [SNIP] > >> > Adding GitHub username support to bugs.python.org > >> > +++++++++++++++++++++++++++++++++++++++++++++++++ > >> > To keep tracking of CLA signing under the direct control of the PSF, > >> > tracking who has signed the PSF CLA will be continued by marking that > >> > fact as part of someone's bugs.python.org user profile. What this > >> > means is that an association will be needed between a person's > >> > bugs.python.org [#b.p.o]_ account and their GitHub account, which > >> > will be done through a new field in a user's profile. > >> > > >> > >> We have to decide how to deal with users that don't have a b.p.o > account. > >> The two options that we discussed in the previous mails are: > >> 1) require them to create a b.p.o account; > >> 2) allow them to log in to b.p.o using their github account; > >> (see also next section) > > > > > > Both require creating an account, it just varies whether they log in > using > > GitHub or not. I don't see how we can avoid that if we are going to > continue > > to own the CLA dataset. Honestly I don't see this as a big issue as we > have > > not seemed to have any issues with people creating accounts up to this > > point. > > > > It's not a big issue, but there will be people that only have a github > account, and we will need to explain them that in order to accept > their PRs we need them to sign the CLA, and in order to sign the CLA > we need them to create a b.p.o account and link it to their github > username. > I expect this to be the common case. I wish there was a way to avoid it, but if we want people to participate on the issue tracker they will need the account anyway. Plus it's no worse than it is today as I expect it's already the case most people have a GItHub account but not a b.p.o one. > > >> > >> > Linking a pull request to an issue > >> > ++++++++++++++++++++++++++++++++++ > >> > An association between a pull request and an issue is needed to track > >> > when a fix has been proposed. The association needs to be many-to-one > >> > as there can take multiple pull requests to solve a single issue > >> > (technically it should be a many-to-many association for when a > >> > single fix solves multiple issues, but this is fairly rare and issues > >> > can be merged into one using the ``Superceder`` field on the issue > >> > tracker). > >> > > >> > Association between a pull request and an issue will be done based on > >> > detecting the regular expression``[Ii]ssue #(?P\d+)``. If > >> > this is specified in either the title or in the body of a message on > >> > a pull request then connection will be made on > >> > bugs.python.org [#b.p.o]_. A label will also be added to the pull > >> > request to signify that the connection was made successfully. This > >> > could lead to incorrect associations if the wrong issue or > >> > referencing another issue was done, but these are rare occasions. > >> > > >> > >> Is there a way to associate the PR to an issue (in case the user > >> forgot) or change association (in case it got the wrong number) after > >> the creation of the PR? > > > > > > You tell me. :) I assume any bot we write to handle this will monitor > > PR-level comments since GitHub doesn't notify on title changes. But we > can > > choose any workflow we want, so if we want it to be an explicit command > like > > `/bot issue 12345` then we can do that instead. > > > > I was asking about the github side. On b.p.o it's not a problem > adding PRs either automatically or manually, but I don't know if the > same can be done from github (it already happens when a commit message > includes the wrong issue number -- the commit can not be changed and > the notification is sent to the wrong issue on b.p.o). > Even if it's not possible, I guess we could still add a comment to > github with the correct issue number, and add the PR to the issue and > the people to the nosy list manually. > So are you asking about a b.p.o -> GH association, so that you can make the association on b.p.o and have it show up somehow? If that's your question then we can add a comment like we do on b.p.o. > [SNIP] > >> > >> > Backup of pull request data > >> > ''''''''''''''''''''''''''' > >> > Since GitHub [#github]_ is going to be used for code hosting and code > >> > review, those two things need to be backed up. In the case of code > >> > hosting, the backup is implicit as all non-shallow Git [#git]_ clones > >> > contain the full history of the repository, hence there will be many > >> > backups of the repository. > >> > > >> > >> If possible I would still prefer an "official" backup. I don't think > >> we want to go around asking who has the latest copy of the repo in > >> case something happens to GitHub. > > > > > > I would be shocked if a core developer doesn't have an up-to-date > repository > > (then again I don't expect to have to flee GitHub overnight, giving us > time > > to update). But if you want to make it this an optional feature then I'm > > fine with that (I don't think the PSF infrastructure team will have issue > > with a regularly updated clone of the repository). > > > > Yes, this is not a particularly realistic issue -- even without an > official backup we won't lose any code. It's mostly to have an > official backup and procedure to restore the repo instead of having to > figure out what to do when it happens (even if it probably never > will). The infra team can decide if this is a reasonable request or > if it's not worth the extra effort. > +1 > [SNIP] > >> > Test coverage report > >> > '''''''''''''''''''' > >> > Getting an up-to-date test coverage report for Python's standard > >> > library would be extremely beneficial as generating such a report can > >> > take quite a while to produce. > >> > > >> > There are a couple pre-existing services that provide free test > >> > coverage for open source projects. Which option is best is an open > >> > issue: `Choosing a test coverage service`_. > >> > > >> > >> Do we want to eventually request that all new code introduced is fully > >> covered by tests? > > > > > > I have always told sprinters that 80% is a good guideline. I wouldn't > ever > > want a rule, though. Obviously lowering coverage would not be great and > > should be avoided, but if > > > > Maybe we could use red/yellow/green labels to indicate the coverage > level. That alone might lead contributors to aim for the green label > without having to create and enforce any rule. > To give you an idea of at least how Coveralls integration looks, https://github.com/python-modernize/python-modernize/pull/117 > > >> > >> I think having an indication of how much code in a PR is covered by > >> tests would be useful regardless of the answer to the previous > >> question. > > > > > > I know Coveralls already does this; don't know about Codecov. > > > >> > >> > >> > > >> > Link web content back to files that it is generated from > >> > '''''''''''''''''''''''''''''''''''''''''''''''''''''''' > >> > It would be helpful for people who find issues with any of the > >> > documentation that is generated from a file to have a link on each > >> > page which points back to the file on GitHub [#github]_ that stores > >> > the content of the page. That would allow for quick pull requests to > >> > fix simple things such as spelling mistakes. > >> > > >> > >> Here you are talking about PEPs/devguide/docs.p.o, right? > > > > > > Yes. > > > > FWIW the docs.python.org pages already have a "report a bug" link in > the sidebar and also in the footer, but they both just redirect to > https://docs.python.org/3/bugs.html . > Yep, which is what made me think that it would be nice if we could direct people directly to the actual page instead of having to figure it out. But then again, if we think a lot of drive-by PRs will be doc-based and from people who have never contributed before, then we will have to wait for them to sign the CLA anyway, so maybe it isn't worth it? I guess the direct link to the underlying content is only useful if people who have signed the CLA will use it the most. > > >> > >> FWIW this was problem was supposed to be fixed with pootle, but that > >> project seems dead (not sure if due to technical reasons, or simply > >> because no one had time to integrate it). > >> > >> > Splitting out parts of the documentation into their own repositories > >> > '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' > >> > While certain parts of the documentation at https://docs.python.org > >> > change with the code, other parts are fairly static and are not > >> > tightly bound to the CPython code itself. The following sections of > >> > the documentation fit this category of slow-changing, > >> > loosely-coupled: > >> > > >> > * `Tutorial `__ > >> > * `Python Setup and Usage > >> > `__ > >> > * `HOWTOs `__ > >> > * `Installing Python Modules > >> > `__ > >> > * `Distributing Python Modules > >> > `__ > >> > * `Extending and Embedding > >> > `__ > >> > * `FAQs `__ > >> > > >> > These parts of the documentation could be broken out into their own > >> > repositories to simplify their maintenance and to expand who has > >> > commit rights to them to ease in their maintenance. > >> > >> I would still consider these somewhat dynamic (especially between Py 2 > >> and Py 3). > >> There are other documents that are more version-independent, such as > >> the whatsnew pages, > > > > > > The What's New docs get updated with changes so that can't be pulled out > > from the cpython repo (especially if my dream ever comes true of making > > people be better about making sure that document gets updated with > changes > > that warrant being mentioned there). > > > > The reason to keep them in a separate repo is that they are identical > copies of the same document. > If you fix a typo in the whatsnew/2.7, you have to fix the same typo > in all branches, and in theory the page should always be identical in > each branch. Sure, but I try to make it a habit to update What's New at the same time as committing something that warrants a mention. Pulling that out will become a bigger chore. Having said that, if we want core developers to be the ones that author those kinds of changes then having it be a separate repo won't necessarily be as critical. We would just probably need to add a "needs What's New mention" label or something to keep track of what needs to be added so people didn't forget. > Howtos, FAQs, etc. might differ between major and even minor versions > as new features are added and old features deprecated, so it makes > sense to have (slightly) different versions in each branch (unless you > want to move them outside the cpython repo and still keep separate > branches). > For example > https://docs.python.org/3/faq/programming.html#is-there-an-equivalent-of-c-s-ternary-operator > had a different answer before 2.5 :) > I suspect that when that kind of situation occurred that I version-specific branch would be created and then merged in once a release went out. > > Also moving them might make things more complicated (from simply > having to find/clone another repo to building docs for docs.python.org > from different sources). > It's possible, but I doubt it. We could have a docs.python.org repo that contains everything but the docs carried in the cpython repo. Or we can have these other doc repos be Git submodules of cpython that you can check out if you want, but it isn't necessary to. What I do know is this idea has come up many times in the past from the perspective of being able to give out commit rights to the docs much more readily than with the cpython repo and I think that's a good idea. > [SNIP] > >> > Git CLI commands for committing a pull request to cpython > >> > --------------------------------------------------------- > >> > Because Git [#git]_ may be a new version control system for core > >> > developers, the commands people are expected to run will need to be > >> > written down. These commands also need to keep a linear history while > >> > giving proper attribution to the pull request author. > >> > > >> > Another set of commands will also be necessary for when working with > >> > a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear > >> > history will be kept implicitly, but it will need to make sure to > >> > keep/add attribution. > >> > > >> > >> Nick Coghlan, Pierre-Yves David (a Mercurial dev), and Shiyao Ma (one > >> of our GSoC student) have been working on an HG extension that > >> simplifies interaction with the bug tracker (see the list of patches, > >> download/apply them, upload new patches): > >> https://bitbucket.org/introom/hg-cpydev > >> In a previous email, someone mentioned an alias that allows an easier > >> interaction with PRs. Would it make sense to write and distribute an > >> official git extension that provides extra commands/aliases for these > >> set of commands? (I guess the answer depends on how many tasks we > >> have and how straightforward it is to do with plain git commands.) > > > > > > It quite possibly might be. Otherwise shell commands could be written and > > kept in Tools/. I major perk IMO with Git over Mercurial is Git Bash > comes > > with GIt and that gives you Bash on Windows. That makes writing > > cross-platform shell scripts to help with this sort of thing easy without > > leaving Windows users stranded. > > > > Isn't that also possible with HG, since extensions are written in Python? > I mean .sh files, not extensions, e.g., we have a shell script in Tools that calls Git directly. > (BTW, is it possible/reasonable to write git extensions/hooks in Python?) > I believe so. From my understanding, Git doesn't go with an API solution like Mercurial and instead prefers a way to simply specify a naming scheme that will call commands on your $PATH. See https://www.atlassian.com/git/articles/extending-git/ as an example. > > >> > >> > >> > How to handle the Misc/NEWS file > >> > -------------------------------- > >> > There are two competing approaches to handling > >> > ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on > >> > bugs.python.org [#b.p.o]_. This would mean an issue that is marked > >> > as "resolved" could not be closed until a news entry is added in the > >> > "news" field in the issue tracker. The benefit of tying the news > >> > entry to the issue is it makes sure that all changes worthy of a news > >> > entry have an accompanying issue. It also makes classifying a news > >> > entry automatic thanks to the Component field of the issue. The > >> > Versions field of the issue also ties the news entry to which Python > >> > releases were affected. A script would be written to query > >> > bugs.python.org for relevant new entries for a release and to produce > >> > the output needed to be checked into the code repository. This > >> > approach is agnostic to whether a commit was done by CLI or bot. > >> > > >> > The competing approach is to use an individual file per news entry, > >> > containg the text for the entry. In this scenario each feature > >> > >> Typo: containing > >> > >> > release would have its own directory for news entries and a separate > >> > file would be created in that directory that was either named after > >> > the issue it closed or a timestamp value (which prevents collisions). > >> > Merges across branches would have no issue as the news entry file > >> > would still be uniqeuely named and in the directory of the latest > >> > >> Typo: uniquely > >> > >> > version that contained the fix. A script would collect all news entry > >> > files no matter what directory they reside in and create an > >> > appropriate news file (the release directory can be ignored as the > >> > mere fact that the file exists is enough to represent that the entry > >> > belongs to the release). Classification can either be done by keyword > >> > in the new entry file itself or by using subdirectories representing > >> > each news entry classification in each release directory (or > >> > classification of news entries could be dropped since critical > >> > information is captured by the "What's New" documents which are > >> > organized). The benefit of this approach is that it keeps the changes > >> > with the code that was actually changed. It also ties the message to > >> > being part of the commit which introduced the change. For a commit > >> > made through the CLI, a script will be provided to help generate the > >> > file. In a bot-driven scenario, the merge bot will have a way to > >> > specify a specific news entry and create the file as part of its > >> > flattened commit (while most likely also supporting using the first > >> > line of the commit message if no specific news entry was specified). > >> > Code for this approach has been written previously for the Mercurial > >> > workflow at http://bugs.python.org/issue18967. There is also a tool > >> > from the community at https://pypi.python.org/pypi/towncrier. > >> > > >> > >> Does using git (and fast-forward merges, rebases, etc.) still create > >> conflicts? > >> (I guess the answer is yes, but perhaps we should double-check.) > > > > > > I believe so, but I guess I could be wrong since I have not explicitly > > tested it. > > > >> > >> > >> If it does, there's also a third option: writing a merge script. > >> I wrote a basic one for hg and it seemed to work decently, perhaps > >> with git it's even easier. > > > > > > I'll mention it, but i suspect the file-based solution will win out > based on > > the feeling I have gotten from people when this topic has come up before. > > > > I was re-reading the issue, and found due interesting links: > 1) > https://mail.python.org/pipermail/python-dev/2014-December/137393.html > (Pierre-Yves David actually convinced me to write the merge script and > helped me) > 2) https://github.com/twisted/newsbuilder (this can be another > option that can be added to the PEP) > Thanks, I'll add them. > > > A few additional points that you might want to add to the PEP: > * A new workflow for releasing Python should also be defined, and PEP > 101 updated accordingly. > I view this as implicit. > * The devguide also needs to be updated. > Implicit, but I can call this out. I assume it will end up with a github-migration branch for a while to store updates until the migration occurs. > * We should decide what to do with all the other repos at hg.python.org They are all personal repos, so people can do what they want with them. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jan 18 13:45:07 2016 From: brett at python.org (Brett Cannon) Date: Mon, 18 Jan 2016 18:45:07 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On Sun, 17 Jan 2016 at 18:58 Chris Angelico wrote: > On Mon, Jan 18, 2016 at 7:48 AM, Brett Cannon wrote: > > Luckily, Git [#git]_ does not require GitHub's workflow and so one can > > be chosen which gives us a linear history by using Git's CLI. The > > expectation is that all pull requests will be fast-forwarded and > > rebased before being pushed to the master repository. This should > > give proper attribution to the pull request author in the Git > > history. > > > > A second set of recommended commands will also be written for > > committing a contribution from a patch file uploaded to > > bugs.python.org [#b.p.o]_. This will obviously help keep the linear > > history, but it will need to be made to have attribution to the patch > > author. > > Point to note: If you want to make use of git's separate author and > committer records (so the author is the person who wrote the original > patch, and the committer is the core dev who actually pushed it), > you'll forfeit the identifiable hashes on commits. Every commit will > have to be rebased (or amended, which is a short-hand form of rebase) > to change its committer, which creates a brand new commit with a new > hash. GitHub won't recognize the push as incorporating the original > commit, and neither will people's tools elsewhere. > > IMO this is a worthwhile trade-off, but it is a cost, and may be worth > mentioning in the PEP. It's no worse than the current state (where a > Mercurial commit completely loses track of its original author, unless > it's mentioned in the human-readable message), but it's perhaps not > what people will be expecting. > I don't quite follow this. If you do a ff + rebase for the final commit how does that affect the hash of the final commit? Or what if you take a patch, apply it, and as part of the `git commit` command you specify the author on the command-line? I understand how it would change things if we were updating a pre-existing Git repository, but I'm only talking about future commits and not necessarily trying to retroactively do this for the direct migration of a repository. -Brett > > (Also, when it comes time to write up the actual commands for people, > it'll need to be made clear that an actual fast-forward isn't > acceptable, as it won't update the committer. Amending or rebasing > MUST be done.) > > ChrisA > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Mon Jan 18 14:03:48 2016 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Mon, 18 Jan 2016 11:03:48 -0800 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: <007572AB-B8CB-4CD5-BDE8-462C35A23F26@gmail.com> > On Jan 18, 2016, at 10:45, Brett Cannon wrote: > > > > On Sun, 17 Jan 2016 at 18:58 Chris Angelico > wrote: > On Mon, Jan 18, 2016 at 7:48 AM, Brett Cannon > wrote: > > Luckily, Git [#git]_ does not require GitHub's workflow and so one can > > be chosen which gives us a linear history by using Git's CLI. The > > expectation is that all pull requests will be fast-forwarded and > > rebased before being pushed to the master repository. This should > > give proper attribution to the pull request author in the Git > > history. > > > > A second set of recommended commands will also be written for > > committing a contribution from a patch file uploaded to > > bugs.python.org [#b.p.o]_. This will obviously help keep the linear > > history, but it will need to be made to have attribution to the patch > > author. > > Point to note: If you want to make use of git's separate author and > committer records (so the author is the person who wrote the original > patch, and the committer is the core dev who actually pushed it), > you'll forfeit the identifiable hashes on commits. Every commit will > have to be rebased (or amended, which is a short-hand form of rebase) > to change its committer, which creates a brand new commit with a new > hash. GitHub won't recognize the push as incorporating the original > commit, and neither will people's tools elsewhere. > > IMO this is a worthwhile trade-off, but it is a cost, and may be worth > mentioning in the PEP. It's no worse than the current state (where a > Mercurial commit completely loses track of its original author, unless > it's mentioned in the human-readable message), but it's perhaps not > what people will be expecting. > > I don't quite follow this. If you do a ff + rebase for the final commit how does that affect the hash of the final commit? Or what if you take a patch, apply it, and as part of the `git commit` command you specify the author on the command-line? I understand how it would change things if we were updating a pre-existing Git repository, but I'm only talking about future commits and not necessarily trying to retroactively do this for the direct migration of a repository. > I think what is ment here is that GitHub has some features that allows you to go from commit number to which PR it originate from. When you rebase to avoid merge commit, you will generate new commit hashes, which will make these functionality not work. You can see an example here[1], for example the the indicator `master (#9124)`, indicate that this commit was merged into master by PR #9124. this is typically useful to get back to the discussion/review of the PR. The other thing you might loose is the auto closing of Pull-requests once the commit. You will lose also other nice things like intermediate CI status on commit, but that?s less interesting that above features. Agreed that might seem not that useful, but greatly improve some tasks from time to time. -- Matthias [1]: https://github.com/ipython/ipython/commit/de68bf5d79f4df17dd8989e0b1a85ed3af6ef330 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jan 18 14:53:29 2016 From: brett at python.org (Brett Cannon) Date: Mon, 18 Jan 2016 19:53:29 +0000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: <007572AB-B8CB-4CD5-BDE8-462C35A23F26@gmail.com> References: <007572AB-B8CB-4CD5-BDE8-462C35A23F26@gmail.com> Message-ID: On Mon, 18 Jan 2016 at 11:03 Matthias Bussonnier < bussonniermatthias at gmail.com> wrote: > On Jan 18, 2016, at 10:45, Brett Cannon wrote: > > > > On Sun, 17 Jan 2016 at 18:58 Chris Angelico wrote: > >> On Mon, Jan 18, 2016 at 7:48 AM, Brett Cannon wrote: >> > Luckily, Git [#git]_ does not require GitHub's workflow and so one can >> > be chosen which gives us a linear history by using Git's CLI. The >> > expectation is that all pull requests will be fast-forwarded and >> > rebased before being pushed to the master repository. This should >> > give proper attribution to the pull request author in the Git >> > history. >> > >> > A second set of recommended commands will also be written for >> > committing a contribution from a patch file uploaded to >> > bugs.python.org [#b.p.o]_. This will obviously help keep the linear >> > history, but it will need to be made to have attribution to the patch >> > author. >> >> Point to note: If you want to make use of git's separate author and >> committer records (so the author is the person who wrote the original >> patch, and the committer is the core dev who actually pushed it), >> you'll forfeit the identifiable hashes on commits. Every commit will >> have to be rebased (or amended, which is a short-hand form of rebase) >> to change its committer, which creates a brand new commit with a new >> hash. GitHub won't recognize the push as incorporating the original >> commit, and neither will people's tools elsewhere. >> >> IMO this is a worthwhile trade-off, but it is a cost, and may be worth >> mentioning in the PEP. It's no worse than the current state (where a >> Mercurial commit completely loses track of its original author, unless >> it's mentioned in the human-readable message), but it's perhaps not >> what people will be expecting. >> > > I don't quite follow this. If you do a ff + rebase for the final commit > how does that affect the hash of the final commit? Or what if you take a > patch, apply it, and as part of the `git commit` command you specify the > author on the command-line? I understand how it would change things if we > were updating a pre-existing Git repository, but I'm only talking about > future commits and not necessarily trying to retroactively do this for the > direct migration of a repository. > > > I think what is meant here is that GitHub has some features that allows > you to go from commit number to which PR it originate from. > When you rebase to avoid merge commit, you will generate new commit > hashes, which will make these functionality not work. > > You can see an example here[1], for example the the indicator > `master (#9124)`, indicate that this commit was merged into master by PR > #9124. > this is typically useful to get back to the discussion/review of the PR. > > The other thing you might lose is the auto closing of Pull-requests once > the commit. > > You will lose also other nice things like intermediate CI status on > commit, but that?s less interesting that above features. > Agreed that might seem not that useful, but greatly improve some tasks > from time to time. > That's all true and a consequence of a linear history. We will probably need to get into the habit of pasting the resulting commit into the PR when it gets closed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Mon Jan 18 15:07:21 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 18 Jan 2016 15:07:21 -0500 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: <007572AB-B8CB-4CD5-BDE8-462C35A23F26@gmail.com> Message-ID: <569D45F9.60408@gmail.com> On 2016-01-18 2:53 PM, Brett Cannon wrote: > > You will lose also other nice things like intermediate CI status > on commit, but that?s less interesting that above features. > Agreed that might seem not that useful, but greatly improve some > tasks from time to time. > > > That's all true and a consequence of a linear history. We will > probably need to get into the habit of pasting the resulting commit > into the PR when it gets closed. FWIW that's how we do it in asyncio repo on github. Yury From rosuav at gmail.com Mon Jan 18 15:07:45 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 19 Jan 2016 07:07:45 +1100 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: Message-ID: On Tue, Jan 19, 2016 at 5:45 AM, Brett Cannon wrote: > I don't quite follow this. If you do a ff + rebase for the final commit how > does that affect the hash of the final commit? Or what if you take a patch, > apply it, and as part of the `git commit` command you specify the author on > the command-line? I understand how it would change things if we were > updating a pre-existing Git repository, but I'm only talking about future > commits and not necessarily trying to retroactively do this for the direct > migration of a repository. Not sure what you mean by a rebase here. There's two possibilities: 1) The PR is based on what's exactly the current branch head. In that case, you can do a fast-forward. 2) The PR is *not* based on the current branch head (because something else got pushed already). A fast-forward is not possible. In case #2, the normal two options are either a merge commit (GitHub's default, and non-linear history), or rebasing the PR's commits on top of the current branch head, which is pretty much a matter of turning them into patch files and applying those patches as if they'd just been committed. This creates new commit hashes with the committer name/email and date set by the person who does the rebase - and then allows them to be fast-forwarded into the main branch. In case #1, though, there is no rebase necessary. You can have linear history by fast-forwarding. But in that case, the committer would be the person who put it onto the PR, not the one who pushed it to the official repo. To ensure that the committer field is *always* updated, you would need to rebase (or amend, aka "mini rebase") the commits, seemingly unnecessary in git terms, but important for repo integrity. It wouldn't be hard to write a pre-receive hook that validates the commits and ensures that everyone listed in the Committer field is, indeed, a valid committer. ChrisA From ncoghlan at gmail.com Mon Jan 18 22:40:26 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Jan 2016 13:40:26 +1000 Subject: [core-workflow] PEP 512: migrating hg.python.org to GitHub In-Reply-To: References: <007572AB-B8CB-4CD5-BDE8-462C35A23F26@gmail.com> Message-ID: On 19 January 2016 at 05:53, Brett Cannon wrote: > > On Mon, 18 Jan 2016 at 11:03 Matthias Bussonnier > wrote: >> I think what is meant here is that GitHub has some features that allows >> you to go from commit number to which PR it originate from. >> When you rebase to avoid merge commit, you will generate new commit >> hashes, which will make these functionality not work. >> >> You can see an example here[1], for example the the indicator `master >> (#9124)`, indicate that this commit was merged into master by PR #9124. >> this is typically useful to get back to the discussion/review of the PR. >> >> The other thing you might lose is the auto closing of Pull-requests once >> the commit. >> >> You will lose also other nice things like intermediate CI status on >> commit, but that?s less interesting that above features. >> Agreed that might seem not that useful, but greatly improve some tasks >> from time to time. > > > That's all true and a consequence of a linear history. We will probably need > to get into the habit of pasting the resulting commit into the PR when it > gets closed. It also falls into the category of problem that a workflow bot addresses - it just becomes a requirement for the bot to comment on and close the original PR with a reference to the commit, in addition to actually doing the commit itself. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Tue Jan 19 19:51:14 2016 From: brett at python.org (Brett Cannon) Date: Wed, 20 Jan 2016 00:51:14 +0000 Subject: [core-workflow] Updated draft of PEP 512 Message-ID: I have tried to take into account all of the corrections and suggestions people have made. If you want to see exactly what has changed since the last post, look at https://github.com/brettcannon/github-transition-pep/blob/master/pep-0512.rst and its commit history. In the last round of discussions, the most divisive thing was the switch to a cherry-picking workflow over the current merge-up one. I tried to flesh out the explanation for the planned change, but if people still want to discuss it then someone who wants to see it kept to a merge-up workflow needs to start a separate thread on that topic. Otherwise my hope is we can agree that the PEP covers what needs to be done soon so that we can start prioritizing work and then finding people to work on various things. And if you do reply to this email, please trim out what you don't comment on as the length of this PEP is such that after one reply the emails start getting held for moderation due to email size. ---------- PEP: 512 Title: Migrating from hg.python.org to GitHub Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Status: Active Type: Process Content-Type: text/x-rst Created: Post-History: 17-Jan-2015, 19-Jan-2015 Abstract ======== This PEP outlines the steps required to migrate Python's development process from Mercurial [#hg]_ as hosted at hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting the minimum goals of this PEP should allow for the development process of Python to be as productive as it currently is, and meeting its extended goals should improve it. Rationale ========= In 2014, it became obvious that Python's custom development process was becoming a hindrance. As an example, for an external contributor to submit a fix for a bug that eventually was committed, the basic steps were: 1. Open an issue for the bug at bugs.python.org [#b.p.o]_. 2. Checkout out the CPython source code from hg.python.org [#h.p.o]_. 3. Make the fix. 4. Upload a patch. 5. Have a core developer review the patch using our fork of the Rietveld code review tool [#rietveld]_. 6. Download the patch to make sure it still applies cleanly. 7. Run the test suite manually. 8. Commit the change manually. 9. If the change was for a bugfix release, merge into the in-development branch. 10. Run the test suite manually again. 11. Commit the merge. 12. Push the changes. This is a very heavy, manual process for core developers. Even in the simple case, you could only possibly skip the code review step, as you would still need to build the documentation. This led to patches languishing on the issue tracker due to core developers not being able to work through the backlog fast enough to keep up with submissions. In turn, that led to a side-effect issue of discouraging outside contribution due to frustration from lack of attention, which is dangerous problem for an open source project as it runs counter to having a viable future for the project. Simply accepting patches uploaded to bugs.python.org [#b.p.o]_ is potentially simple for an external contributor, it is as slow and burdensome as it gets for a core developer to work with. Hence the decision was made in late 2014 that a move to a new development process was needed. A request for PEPs proposing new workflows was made, in the end leading to two: PEP 481 and PEP 507 proposing GitHub [#github]_ and GitLab [#gitlab]_, respectively. The year 2015 was spent off-and-on working on those proposals and trying to tease out details of what made them different from each other on the core-workflow mailing list [#core-workflow]_. PyCon US 2015 also showed that the community was a bit frustrated with our process due to both cognitive overhead for new contributors and how long it was taking for core developers to look at a patch (see the end of Guido van Rossum's keynote at PyCon US 2015 [#guido-keynote]_ as an example of the frustration). On January 1, 2016, the decision was made by Brett Cannon to move the development process to GitHub. The key reasons for choosing GitHub were [#reasons]_: * Maintaining custom infrastructure has been a burden on volunteers (e.g., a custom fork of Rietveld [#rietveld]_ that is not being maintained is currently being used). * The custom workflow is very time-consuming for core developers (not enough automated tooling built to help support it). * The custom workflow is a hindrance to external contributors (acts as a barrier of entry due to time required to ramp up on development process). * There is no feature differentiating GitLab from GitHub beyond GitLab being open source. * Familiarity with GitHub is far higher amongst core developers and external contributors than with GitLab. * Our BDFL prefers GitHub (who would be the first person to tell you that his opinion shouldn't matter, but the person making the decision felt it was important that the BDFL feel comfortable with the workflow of his own programming language to encourage his continued participation). There's even already an unofficial image to use to represent the migration to GitHub [#pythocat]_. The overarching goal of this migration is to improve the development process to the extent that a core developer can go from external contribution submission through all the steps leading to committing said contribution all from within a browser on a tablet with WiFi using *some* development process (this does not inherently mean GitHub's default workflow). All of this will be done in such a way that if an external contributor chooses not to use GitHub then they will continue to have that option. Repositories to Migrate ======================= While hg.python.org [#h.p.o]_ hosts many repositories, there are only five key repositories that should move: 1. devinabox [#devinabox-repo]_ 2. benchmarks [#benchmarks-repo]_ 4. peps [#peps-repo]_ 5. devguide [#devguide-repo]_ 6. cpython [#cpython-repo]_ The devinabox and benchmarks repositories are code-only. The peps and devguide repositories involve the generation of webpages. And the cpython repository has special requirements for integration with bugs.python.org [#b.p.o]_. Migration Plan ============== The migration plan is separated into sections based on what is required to migrate the repositories listed in the `Repositories to Migrate`_ section. Completion of requirements outlined in each section should unblock the migration of the related repositories. The sections are expected to be completed in order, but not necessarily the requirements within a section. Requirements for Code-Only Repositories --------------------------------------- Completion of the requirements in this section will allow the devinabox and benchmarks repositories to move to GitHub. While devinabox has a sufficiently descriptive name, the benchmarks repository does not; therefore, it will be named "python-benchmark-suite". Create a 'python-dev' team '''''''''''''''''''''''''' To manage permissions, a 'python-dev' team will be created as part of the python organization [#github-python-org]_. Any repository that is moved will have the 'python-dev' team added to it with write permissions [#github-org-perms]_. Anyone who previously had rights to manage SSH keys on hg.python.org will become a team maintainer for the 'python-dev' team. Define commands to move a Mercurial repository to Git ''''''''''''''''''''''''''''''''''''''''''''''''''''' Since moving to GitHub also entails moving to Git [#git]_, we must decide what tools and commands we will run to translate a Mercurial repository to Git. The exact tools and steps to use are an open issue; see `Tools and commands to move from Mercurial to Git`_. CLA enforcement ''''''''''''''' A key part of any open source project is making sure that its source code can be properly licensed. This requires making sure all people making contributions have signed a contributor license agreement (CLA) [#cla]_. Up until now, enforcement of CLA signing of contributed code has been enforced by core developers checking whether someone had an ``*`` by their username on bugs.python.org [#b.p.o]_. With this migration, the plan is to start off with automated checking and enforcement of contributors signing the CLA. Adding GitHub username support to bugs.python.org +++++++++++++++++++++++++++++++++++++++++++++++++ To keep tracking of CLA signing under the direct control of the PSF, tracking who has signed the PSF CLA will be continued by marking that fact as part of someone's bugs.python.org user profile. What this means is that an association will be needed between a person's bugs.python.org [#b.p.o]_ account and their GitHub account, which will be done through a new field in a user's profile. This does implicitly require that contributors will need both a GitHub [#github]_ and bugs.python.org account in order to sign the CLA and contribute through GitHub. A bot to enforce CLA signing ++++++++++++++++++++++++++++ With an association between someone's GitHub account and their bugs.python.org [#b.p.o]_ account, which has the data as to whether someone has signed the CLA, a bot can monitor pull requests on GitHub and denote whether the contributor has signed the CLA. If the user has signed the CLA, the bot will add a positive label to the issue to denote the pull request has no CLA issues (e.g., a green label stating, "CLA: ?"). If the contributor has not signed a CLA, a negative label will be added to the pull request will be blocked using GitHub's status API (e.g., a red label stating, "CLA: ?"). If a contributor lacks a bugs.python.org account, that will lead to another label (e.g., "CLA: ? (no account)"). Using a label for both positive and negative cases provides a fallback notification if the bot happens to fail, preventing potential false-positives or false-negatives. It also allows for an easy way to trigger the bot again by simply removing a CLA-related label. Requirements for Web-Related Repositories ----------------------------------------- Due to their use for generating webpages, the devguide [#devguide-repo]_ and peps [#peps-repo]_ repositories need their respective processes updated to pull from their new Git repositories. The devguide repository might also need to be named ``python-devguide`` to make sure the repository is not ambiguous when viewed in isolation from the python organization [#github-python-org]_. Requirements for the cpython Repository --------------------------------------- Obviously the most active and important repository currently hosted at hg.python.org [#h.p.o]_ is the cpython repository [#cpython-repo]_. Because of its importance and high- frequency use, it requires more tooling before being moved to GitHub compared to the other repositories mentioned in this PEP. Document steps to commit a pull request ''''''''''''''''''''''''''''''''''''''' During the process of choosing a new development workflow, it was decided that a linear history is desired. People preferred having a single commit representing a single change instead of having a set of unrelated commits lead to a merge commit that represented a single change. This means that the convenient "Merge" button in GitHub pull requests is undesirable, as it creates a merge commit along with all of the contributor's individual commits (this does not affect the other repositories where the desire for a linear history doesn't exist). Luckily, Git [#git]_ does not require GitHub's workflow and so one can be chosen which gives us a linear history by using Git's CLI. The expectation is that all pull requests will be fast-forwarded and rebased before being pushed to the master repository. This should give proper attribution to the pull request author in the Git history. This does have the consequence of losing some GitHub features such as automatic closing of pull requests, link generation, etc. A second set of recommended commands will also be written for committing a contribution from a patch file uploaded to bugs.python.org [#b.p.o]_. This will obviously help keep the linear history, but it will need to be made to have attribution to the patch author. The exact sequence of commands that will be given as guidelines to core developers is an open issue: `Git CLI commands for committing a pull request to cpython`_. Handling Misc/NEWS '''''''''''''''''' Traditionally the ``Misc/NEWS`` file [#news-file]_ has been problematic for changes which spanned Python releases. Often times there will be merge conflicts when committing a change between e.g., 3.5 and 3.6 only in the ``Misc/NEWS`` file. It's so common, in fact, that the example instructions in the devguide explicitly mention how to resolve conflicts in the ``Misc/NEWS`` file [#devguide-merge-across-branches]_. As part of our tool modernization, working with the ``Misc/NEWS`` file will be simplified. There are currently two competing approaches to solving the ``Misc/NEWS`` problem which are discussed in an open issue: `How to handle the Misc/NEWS file`_. Handling Misc/ACKS '''''''''''''''''' Traditionally the ``Misc/ACKS`` file [#acks-file]_ has been managed by hand. But thanks to Git supporting an ``author`` value as well as a ``committer`` value per commit, authorship of a commit can be part of the history of the code itself. As such, manual management of ``Misc/ACKS`` will become optional. A script will be written that will collect all author and committer names and merge them into ``Misc/ACKS`` with all of the names listed prior to the move to Git. Running this script will become part of the release process. Linking pull requests to issues ''''''''''''''''''''''''''''''' Historically, external contributions were attached to an issue on bugs.python.org [#b.p.o]_ thanks to the fact that all external contributions were uploaded as a file. For changes committed by a core developer who committed a change directly, the specifying of an issue number in the commit message of the format ``Issue #`` at the start of the message led to a comment being posted to the issue linking to the commit. Linking a pull request to an issue ++++++++++++++++++++++++++++++++++ An association between a pull request and an issue is needed to track when a fix has been proposed. The association needs to be many-to-one as there can take multiple pull requests to solve a single issue (technically it should be a many-to-many association for when a single fix solves multiple issues, but this is fairly rare and issues can be merged into one using the ``Superseder`` field on the issue tracker). Association between a pull request and an issue will be done based on detecting the regular expression``[Ii]ssue #(?P\d+)``. If this is specified in either the title or in the body of a message on a pull request then connection will be made on bugs.python.org [#b.p.o]_. A label will also be added to the pull request to signify that the connection was made successfully. This could lead to incorrect associations if the wrong issue or referencing another issue was done, but these are rare occasions. Notify the issue if the pull request is committed +++++++++++++++++++++++++++++++++++++++++++++++++ Once a pull request is closed (merged or not), the issue should be updated to reflect this fact. Update linking service for mapping commit IDs to URLs ''''''''''''''''''''''''''''''''''''''''''''''''''''' Currently you can use https://hg.python.org/lookup/ with a revision ID from either the Subversion or Mercurial copies of the cpython repo [#cpython-repo]_ to get redirected to the URL for that revision in the Mercurial repository. The URL rewriter will need to be updated to redirect to the Git repository and to support the new revision IDs created for the Git repository. Create https://git.python.org ''''''''''''''''''''''''''''' Just as hg.python.org [#h.p.o]_ currently points to the Mercurial repository for Python, git.python.org should do the equivalent for the Git repository. Backup of pull request data ''''''''''''''''''''''''''' Since GitHub [#github]_ is going to be used for code hosting and code review, those two things need to be backed up. In the case of code hosting, the backup is implicit as all non-shallow Git [#git]_ clones contain the full history of the repository, hence there will be many backups of the repository. The code review history does not have the same implicit backup mechanism as the repository itself. That means a daily backup of code review history should be done so that it is not lost in case of any issues with GitHub. It also helps guarantee that a migration from GitHub to some other code review system is feasible were GitHub to disappear overnight. Change sys._mercurial ''''''''''''''''''''' Once Python is no longer kept in Mercurial, the ``sys._mercurial`` attribute will need to be removed. An equivalent ``sys._git`` attribute will be needed to take its place. Update the devguide ''''''''''''''''''' The devguide will need to be updated with details of the new workflow. Mostly likely work will take place in a separate branch until the migration actually occurs. Update PEP 101 '''''''''''''' The release process will need to be updated as necessary. Optional, Planned Features -------------------------- Once the cpython repository [#cpython-repo]_ is migrated, all repositories will have been moved to GitHub [#github]_ and the development process should be on equal footing as before. But a key reason for this migration is to improve the development process, making it better than it has ever been. This section outlines some plans on how to improve things. It should be mentioned that overall feature planning for bugs.python.org [#b.p.o]_ -- which includes plans independent of this migration -- are tracked on their own wiki page [#tracker-plans]_. Bot to handle pull request merging '''''''''''''''''''''''''''''''''' As stated in the section entitled "`Document steps to commit a pull request`_", the desire is to maintain a linear history for cpython. Unfortunately, Github's [#github]_ web-based workflow does not support a linear history. Because of this, a bot should be written to substitute for GitHub's in-browser commit abilities. To start, the bot should accept commands to commit a pull request against a list of branches. This allows for committing a pull request that fixes a bug in multiple versions of Python. More advanced features such as a commit queue can come later. This would linearly apply accepted pull requests and verify that the commits did not interfere with each other by running the test suite and backing out commits if the test run failed. To help facilitate the speed of testing, all patches committed since the last test run can be applied and run in a single test run as the optimistic assumption is that the patches will work in tandem. Inspiration or basis of the bot could be taken from pre-existig bots such as Homu [#homu]_ or Zuul [#zuul]_. The name given to this bot in order to give it commands is an open issue: `Naming the commit bot`_. Continuous integration per pull request ''''''''''''''''''''''''''''''''''''''' To help speed up pull request approvals, continuous integration testing should be used. This helps mitigate the need for a core developer to download a patch simply to run the test suite against the patch. Which free CI service to use is an open issue: `Choosing a CI service`_. Test coverage report '''''''''''''''''''' Getting an up-to-date test coverage report for Python's standard library would be extremely beneficial as generating such a report can take quite a while to produce. There are a couple pre-existing services that provide free test coverage for open source projects. Which option is best is an open issue: `Choosing a test coverage service`_. Notifying issues of pull request comments ''''''''''''''''''''''''''''''''''''''''' The current development process does not include notifying an issue on bugs.python.org [#b.p.o]_ when a review comment is left on Rietveld [#rietveld]_. It would be nice to fix this so that people can subscribe only to comments at bugs.python.org and not GitHub [#github]_ and yet still know when something occurs on GitHub in terms of review comments on relevant pull requests. Current thinking is to post a comment to bugs.python.org to the relevant issue when at least one review comment has been made over a certain period of time (e.g., 15 or 30 minutes). This keeps the email volume down for those that receive both GitHub and bugs.python.org email notifications while still making sure that those only following bugs.python.org know when there might be a review comment to address. Allow bugs.python.org to use GitHub as a login provider ''''''''''''''''''''''''''''''''''''''''''''''''''''''' As of right now, bugs.python.org [#b.p.o]_ allows people to log in using Google, Launchpad, or OpenID credentials. It would be good to expand this to GitHub credentials. Web hooks for re-generating web content ''''''''''''''''''''''''''''''''''''''' The content at https://docs.python.org/, https://docs.python.org/devguide, and https://www.python.org/dev/peps/ are all derived from files kept in one of the repositories to be moved as part of this migration. As such, it would be nice to set up appropriate webhooks to trigger rebuilding the appropriate web content when the files they are based on change instead of having to wait for, e.g., a cronjob to trigger. Link web content back to files that it is generated from '''''''''''''''''''''''''''''''''''''''''''''''''''''''' It would be helpful for people who find issues with any of the documentation that is generated from a file to have a link on each page which points back to the file on GitHub [#github]_ that stores the content of the page. That would allow for quick pull requests to fix simple things such as spelling mistakes. Splitting out parts of the documentation into their own repositories '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' While certain parts of the documentation at https://docs.python.org change with the code, other parts are fairly static and are not tightly bound to the CPython code itself. The following sections of the documentation fit this category of slow-changing, loosely-coupled: * `Tutorial `__ * `Python Setup and Usage `__ * `HOWTOs `__ * `Installing Python Modules < https://docs.python.org/3/installing/index.html>`__ * `Distributing Python Modules < https://docs.python.org/3/distributing/index.html>`__ * `Extending and Embedding `__ * `FAQs `__ These parts of the documentation could be broken out into their own repositories to simplify their maintenance and to expand who has commit rights to them to ease in their maintenance. It has also been suggested to split out the `What's New `__ documents. That would require deciding whether a workflow could be developed where it would be difficult to forget to update What's New (potentially through a label added to PRs, like "What's New needed"). Backup of Git repositories '''''''''''''''''''''''''' While not necessary, it would be good to have official backups of the various Git repositories for disaster protection. It will be up to the PSF infrastructure committee to decide if this is worthwhile or unnecessary. Status ====== Requirements for migrating the devinabox [#devinabox-repo]_ and benchmarks [#benchmarks-repo]_ repositories: * Not started - `Create a 'python-dev' team`_ - `Define commands to move a Mercurial repository to Git`_ - `Adding GitHub username support to bugs.python.org`_ - `A bot to enforce CLA signing`_ * In progress - None * Completed - None Repositories whose build steps need updating: * Not started - peps [#peps-repo]_ - devguide [#devguide-repo]_ * In progress - None * Completed - None Requirements to move over the cpython repo [#cpython-repo]_: * Not started - `Document steps to commit a pull request`_ - `Handling Misc/NEWS`_ - `Handling Misc/ACKS`_ - `Linking a pull request to an issue`_ - `Notify the issue if the pull request is committed`_ - `Update linking service for mapping commit IDs to URLs`_ - `Create https://git.python.org`_ - `Backup of pull request data`_ - `Change sys._mercurial`_ - `Update the devguide`_ - `Update PEP 101`_ * In progress - None * Completed - None Optional features: * Not started - `Bot to handle pull request merging`_ - `Continuous integration per pull request`_ - `Test coverage report`_ - `Notifying issues of pull request comments`_ - `Allow bugs.python.org to use GitHub as a login provider`_ - `Web hooks for re-generating web content`_ - `Link web content back to files that it is generated from`_ - `Splitting out parts of the documentation into their own repositories`_ - `Backup of Git repositories`_ * In progress - None * Completed - None Open Issues =========== For this PEP, open issues are ones where a decision needs to be made to how to approach or solve a problem. Open issues do not entail coordination issues such as who is going to write a certain bit of code. The fate of hg.python.org ------------------------- With the code repositories moving over to Git [#git]_, there is no technical need to keep hg.python.org [#h.p.o]_ running. Having said that, some in the community would like to have it stay functioning as a Mercurial [#hg]_ mirror of the Git repositories. Others have said that they still want a mirror, but one using Git. As maintaining hg.python.org is not necessary, it will be up to the PSF infrastructure committee to decide if they want to spend the time and resources to keep it running. They may also choose whether they want to host a Git mirror on PSF infrastructure. Depending on the decision reached, other ancillary repositories will either be forced to migration or they can choose to simply stay on hg.python.org. Tools and commands to move from Mercurial to Git ------------------------------------------------ A decision needs to be made on exactly what tooling and what commands involving those tools will be used to convert a Mercurial repository to Git. Currently a suggestion has been made to use https://github.com/frej/fast-export. Another suggestion is to use https://github.com/felipec/git-remote-hg. Finally, http://hg-git.github.io/ has been suggested. Git CLI commands for committing a pull request to cpython --------------------------------------------------------- Because Git [#git]_ may be a new version control system for core developers, the commands people are expected to run will need to be written down. These commands also need to keep a linear history while giving proper attribution to the pull request author. Another set of commands will also be necessary for when working with a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear history will be kept implicitly, but it will need to make sure to keep/add attribution. How to handle the Misc/NEWS file -------------------------------- There are three competing approaches to handling ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on bugs.python.org [#b.p.o]_. This would mean an issue that is marked as "resolved" could not be closed until a news entry is added in the "news" field in the issue tracker. The benefit of tying the news entry to the issue is it makes sure that all changes worthy of a news entry have an accompanying issue. It also makes classifying a news entry automatic thanks to the Component field of the issue. The Versions field of the issue also ties the news entry to which Python releases were affected. A script would be written to query bugs.python.org for relevant new entries for a release and to produce the output needed to be checked into the code repository. This approach is agnostic to whether a commit was done by CLI or bot. A competing approach is to use an individual file per news entry, containing the text for the entry. In this scenario each feature release would have its own directory for news entries and a separate file would be created in that directory that was either named after the issue it closed or a timestamp value (which prevents collisions). Merges across branches would have no issue as the news entry file would still be uniquely named and in the directory of the latest version that contained the fix. A script would collect all news entry files no matter what directory they reside in and create an appropriate news file (the release directory can be ignored as the mere fact that the file exists is enough to represent that the entry belongs to the release). Classification can either be done by keyword in the new entry file itself or by using subdirectories representing each news entry classification in each release directory (or classification of news entries could be dropped since critical information is captured by the "What's New" documents which are organized). The benefit of this approach is that it keeps the changes with the code that was actually changed. It also ties the message to being part of the commit which introduced the change. For a commit made through the CLI, a script will be provided to help generate the file. In a bot-driven scenario, the merge bot will have a way to specify a specific news entry and create the file as part of its flattened commit (while most likely also supporting using the first line of the commit message if no specific news entry was specified). Code for this approach has been written previously for the Mercurial workflow at http://bugs.python.org/issue18967. There is also tools from the community like https://pypi.python.org/pypi/towncrier and https://github.com/twisted/newsbuilder . A yet third option is a merge script to handle the conflicts. This approach allows for keeping the NEWS file as a single file. It does run the risk, though, of failure and thus blocking a commit until it can be manually resolved. Naming the commit bot --------------------- As naming things can lead to bikeshedding of epic proportions, Brett Cannon will choose the final name of the commit bot (the name of the project for the bot itself can be anything, this is purely for the name used in giving commands to the bot). The name will come from Monty Python, which is only fitting since Python is named after the comedy troupe. It will most likely come from 'Monty Python and the Holy Grail' [#holy-grail]_ (which happens to be how Brett was introduced to Monty Python). Current ideas on the name include: "Black Knight" sketch [#black-knight-sketch]_: * black-knight * none-shall-pass * just-a-flesh-wound "Bridge of Death" sketch [#bridge-of-death-sketch]_: * bridge-keeper * man-from-scene-24 * five-questions * what-is-your-quest * blue-no-green * air-speed-velocity * your-favourite-colour (and that specific spelling; Monty Python is British, after all) "Killer rabbit" sketch [#killer-rabbit-sketch]_: * killer-rabbit * holy-hand-grenade * 5-is-right-out "Witch Village" sketch [#witch-village-sketch]_: * made-of-wood * burn-her "French Taunter" sketch [#french-taunter-sketch]_: * elderberries * kanigget "Constitutional Peasants" sketch [#constitutional-peasants-sketch]_: * dennis * from-the-masses "Knights Who Say Ni" sketch [#ni-sketch]_: * shubbery * ni >From "Monty Python and the Holy Grail" in general: * brave-sir-robin Choosing a CI service --------------------- There are various CI services that provide free support for open source projects hosted on GitHub [#github]_. Two such examples are Travis [#travis]_ and Codeship [#codeship]_. Whatever solution is chosen will need to not time out in the time it takes to execute Python's test suite. It should optimally provide access to multiple C compilers for more thorough testing. Network access is also beneficial. The current CI service for Python is Pypatcher [#pypatcher]_. A request can be made in IRC to try a patch from bugs.python.org [#b.p.o]_. The results can be viewed at https://ci.centos.org/job/cPython-build-patch/ . Choosing a test coverage service -------------------------------- Getting basic test coverage of Python's standard library can be created simply by using coverage.py [#coverage]_. Getting thorough test coverage is actually quite tricky, with the details outlined in the devinabox's README [#devinabox-repo]_. It would be best if a service could be found that would allow for thorough test coverage, but it might not be feasible. Free test coverage services include Coveralls [#coveralls]_ and Codecov [#codecov]_. Rejected Ideas ============== Separate Python 2 and Python 3 repositories ------------------------------------------- It was discussed whether separate repositories for Python 2 and Python 3 were desired. The thinking was that this would shrink the overall repository size which benefits people with slow Internet connections or small bandwidth caps. In the end it was decided that it was easier logistically to simply keep all of CPython's history in a single repository. Commit multi-release changes in bugfix branch first --------------------------------------------------- As the current development process has changes committed in the oldest branch first and then merged up to the default branch, the question came up as to whether this workflow should be perpetuated. In the end it was decided that committing in the newest branch and then cherry-picking changes into older branches would work best as most people will instinctively work off the newest branch and it is a more common workflow when using Git [#git]_. Cherry-picking is also more bot-friendly for an in-browser workflow. In the merge-up scenario, if you were to request a bot to do a merge and it failed, then you would have to make sure to immediately solve the merge conflicts if you still allowed the main commit, else you would need to postpone the entire commit until all merges could be handled. With a cherry-picking workflow, the main commit could proceed while postponing the merge-failing cherry-picks. This allows for possibly distributing the work of managing conflicting merges. Deriving ``Misc/NEWS`` from the commit logs ------------------------------------------- As part of the discussion surrounding `Handling Misc/NEWS`_, the suggestion has come up of deriving the file from the commit logs itself. In this scenario, the first line of a commit message would be taken to represent the news entry for the change. Some heuristic to tie in whether a change warranted a news entry would be used, e.g., whether an issue number is listed. This idea has been rejected due to some core developers preferring to write a news entry separate from the commit message. The argument is the first line of a commit message compared to that of a news entry have different requirements in terms of brevity, what should be said, etc. References ========== .. [#h.p.o] https://hg.python.org .. [#GitHub] GitHub (https://github.com) .. [#hg] Mercurial (https://www.mercurial-scm.org/) .. [#git] Git (https://git-scm.com/) .. [#b.p.o] https://bugs.python.org .. [#rietveld] Rietveld (https://github.com/rietveld-codereview/rietveld) .. [#gitlab] GitLab (https://about.gitlab.com/) .. [#core-workflow] core-workflow mailing list ( https://mail.python.org/mailman/listinfo/core-workflow) .. [#guido-keynote] Guido van Rossum's keynote at PyCon US ( https://www.youtube.com/watch?v=G-uKNd5TSBw) .. [#reasons] Email to core-workflow outlining reasons why GitHub was selected (https://mail.python.org/pipermail/core-workflow/2016-January/000345.html ) .. [#benchmarks-repo] Mercurial repository for the Unified Benchmark Suite (https://hg.python.org/benchmarks/) .. [#devinabox-repo] Mercurial repository for devinabox ( https://hg.python.org/devinabox/) .. [#peps-repo] Mercurial repository of the Python Enhancement Proposals ( https://hg.python.org/peps/) .. [#devguide-repo] Mercurial repository for the Python Developer's Guide ( https://hg.python.org/devguide/) .. [#cpython-repo] Mercurial repository for CPython ( https://hg.python.org/cpython/) .. [#github-python-org] Python organization on GitHub ( https://github.com/python) .. [#github-org-perms] GitHub repository permission levels ( https://help.github.com/enterprise/2.4/user/articles/repository-permission-levels-for-an-organization/ ) .. [#cla] Python Software Foundation Contributor Agreement ( https://www.python.org/psf/contrib/contrib-form/) .. [#news-file] ``Misc/NEWS`` ( https://hg.python.org/cpython/file/default/Misc/NEWS) .. [#acks-file] ``Misc/ACKS`` ( https://hg.python.org/cpython/file/default/Misc/ACKS) .. [#devguide-merge-across-branches] Devguide instructions on how to merge across branches ( https://docs.python.org/devguide/committing.html#merging-between-different-branches-within-the-same-major-version ) .. [#pythocat] Pythocat (https://octodex.github.com/pythocat/) .. [#tracker-plans] Wiki page for bugs.python.org feature development (https://wiki.python.org/moin/TrackerDevelopmentPlanning) .. [#black-knight-sketch] The "Black Knight" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=dhRUe-gz690) .. [#bridge-of-death-sketch] The "Bridge of Death" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=cV0tCphFMr8) .. [#holy-grail] "Monty Python and the Holy Grail" sketches (https://www.youtube.com/playlist?list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp ) .. [#killer-rabbit-sketch] "Killer rabbit" sketch from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=Nvs5pqf-DMA&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=11 ) .. [#witch-village-sketch] "Witch Village" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=k3jt5ibfRzw&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=12 ) .. [#french-taunter-sketch] "French Taunter" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=A8yjNbcKkNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=13 ) .. [#constitutional-peasants-sketch] "Constitutional Peasants" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=JvKIWjnEPNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=14 ) .. [#ni-sketch] "Knights Who Say Ni" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=zIV4poUZAQo&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=15 ) .. [#homu] Homu (http://homu.io/) .. [#zuul] Zuul (http://docs.openstack.org/infra/zuul/) .. [#travis] Travis (https://travis-ci.org/) .. [#codeship] Codeship (https://codeship.com/) .. [#coverage] coverage.py (https://pypi.python.org/pypi/coverage) .. [#coveralls] Coveralls (https://coveralls.io/) .. [#codecov] Codecov (https://codecov.io/) .. [#pypatcher] Pypatcher (https://github.com/kushaldas/pypatcher) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From darjus at gmail.com Tue Jan 19 22:37:37 2016 From: darjus at gmail.com (Darjus Loktevic) Date: Wed, 20 Jan 2016 03:37:37 +0000 Subject: [core-workflow] PEP 512 - Choosing a CI service Message-ID: Quick background: A little while ago I've added TravisCI integration to Jython github mirror. Most recently one of Jython contributors asked whether we can integrate the xunit XML output of Jython's regrtest with TravisCI. The answer was no, which led me to researching other options. *Why not TravisCI?* TravisCI does not support build artifacts. As simple as that, there's no builtin support for them. After researching for a while, they promised to add it but have no public timelines. This is quite limiting in my view as it would be nice to have more than just a binary pass/fail and manually dig through regrtest output. One of the positives for TravisCI is Mac OS X support. https://travis-ci.org/jythontools/jython *Codeship* I have not actually used it, but from reading documentation it feels more like a deployment service and does not have builtin support for xunit. *Shippable* Configuration format is super similar to TravisCI, which made it easy to try. Has support for both code coverage and xunit test results. Simple UI. One big negative for me with it is that it is slow, and by slow i mean over 1h versus 25min for TravisCI. Presumably paid version is faster. https://app.shippable.com/builds/569eebaed3a5e70d00b309c5 *CircleCI* - My preferred Very fast, free tier gives 4 parallel builds (free tier), supports xunit, supports Mac OS X (have not tried), and supports debugging via SSH (which i think is a cool feature for those cases when it works on your machine and not on the build server and there are no clues as to why). Supports exporting coverage to several coverage services. On the negatives, different configuration format to TravisCI, UI could be simpler/easier. https://circleci.com/gh/darjus/jython/3#tests Cheers, Darjus -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Jan 20 11:04:17 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 20 Jan 2016 10:04:17 -0600 Subject: [core-workflow] PEP 512 - Choosing a CI service In-Reply-To: References: Message-ID: <50BE5D05-B87A-42F4-9762-8F63091EAB88@gmail.com> On January 19, 2016 9:37:37 PM CST, Darjus Loktevic wrote: >Quick background: A little while ago I've added TravisCI integration to >Jython github mirror. Most recently one of Jython contributors asked >whether we can integrate the xunit XML output of Jython's regrtest with >TravisCI. The answer was no, which led me to researching other options. > >*Why not TravisCI?* >TravisCI does not support build artifacts. As simple as that, there's >no >builtin support for them. After researching for a while, they promised >to >add it but have no public timelines. This is quite limiting in my view >as >it would be nice to have more than just a binary pass/fail and manually >dig >through regrtest output. Yes it does! https://docs.travis-ci.com/user/uploading-artifacts/ It even supports putting releases on GitHub: https://docs.travis-ci.com/user/deployment/releases and deploying to various other services: https://docs.travis-ci.com/user/deployment/ >One of the positives for TravisCI is Mac OS X support. >https://travis-ci.org/jythontools/jython > >*Codeship* >I have not actually used it, but from reading documentation it feels >more >like a deployment service and does not have builtin support for xunit. > >*Shippable* >Configuration format is super similar to TravisCI, which made it easy >to >try. Has support for both code coverage and xunit test results. Simple >UI. >One big negative for me with it is that it is slow, and by slow i mean >over >1h versus 25min for TravisCI. Presumably paid version is faster. >https://app.shippable.com/builds/569eebaed3a5e70d00b309c5 > >*CircleCI* - My preferred >Very fast, free tier gives 4 parallel builds (free tier), supports >xunit, >supports Mac OS X (have not tried), and supports debugging via SSH >(which i >think is a cool feature for those cases when it works on your machine >and >not on the build server and there are no clues as to why). Supports >exporting coverage to several coverage services. >On the negatives, different configuration format to TravisCI, UI could >be >simpler/easier. >https://circleci.com/gh/darjus/jython/3#tests > >Cheers, >Darjus > > >------------------------------------------------------------------------ > >_______________________________________________ >core-workflow mailing list >core-workflow at python.org >https://mail.python.org/mailman/listinfo/core-workflow >This list is governed by the PSF Code of Conduct: >https://www.python.org/psf/codeofconduct -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From darjus at gmail.com Wed Jan 20 15:06:45 2016 From: darjus at gmail.com (Darjus Loktevic) Date: Wed, 20 Jan 2016 20:06:45 +0000 Subject: [core-workflow] PEP 512 - Choosing a CI service In-Reply-To: <50BE5D05-B87A-42F4-9762-8F63091EAB88@gmail.com> References: <50BE5D05-B87A-42F4-9762-8F63091EAB88@gmail.com> Message-ID: Hey Ryan, You're right that Travis supports uploading/deploying to various services. My wording is really bad there. For my purposes of exposing the xunit output, i wished that Travis would at least keep/expose some artifacts itself to save me trouble. Do you know if that is supported by Travis? Thanks, Darjus On Thu, Jan 21, 2016 at 3:04 AM Ryan Gonzalez wrote: > > > On January 19, 2016 9:37:37 PM CST, Darjus Loktevic > wrote: > >Quick background: A little while ago I've added TravisCI integration to > >Jython github mirror. Most recently one of Jython contributors asked > >whether we can integrate the xunit XML output of Jython's regrtest with > >TravisCI. The answer was no, which led me to researching other options. > > > >*Why not TravisCI?* > >TravisCI does not support build artifacts. As simple as that, there's > >no > >builtin support for them. After researching for a while, they promised > >to > >add it but have no public timelines. This is quite limiting in my view > >as > >it would be nice to have more than just a binary pass/fail and manually > >dig > >through regrtest output. > > Yes it does! > > https://docs.travis-ci.com/user/uploading-artifacts/ > > It even supports putting releases on GitHub: > > https://docs.travis-ci.com/user/deployment/releases > > and deploying to various other services: > > https://docs.travis-ci.com/user/deployment/ > > >One of the positives for TravisCI is Mac OS X support. > >https://travis-ci.org/jythontools/jython > > > >*Codeship* > >I have not actually used it, but from reading documentation it feels > >more > >like a deployment service and does not have builtin support for xunit. > > > >*Shippable* > >Configuration format is super similar to TravisCI, which made it easy > >to > >try. Has support for both code coverage and xunit test results. Simple > >UI. > >One big negative for me with it is that it is slow, and by slow i mean > >over > >1h versus 25min for TravisCI. Presumably paid version is faster. > >https://app.shippable.com/builds/569eebaed3a5e70d00b309c5 > > > >*CircleCI* - My preferred > >Very fast, free tier gives 4 parallel builds (free tier), supports > >xunit, > >supports Mac OS X (have not tried), and supports debugging via SSH > >(which i > >think is a cool feature for those cases when it works on your machine > >and > >not on the build server and there are no clues as to why). Supports > >exporting coverage to several coverage services. > >On the negatives, different configuration format to TravisCI, UI could > >be > >simpler/easier. > >https://circleci.com/gh/darjus/jython/3#tests > > > >Cheers, > >Darjus > > > > > >------------------------------------------------------------------------ > > > >_______________________________________________ > >core-workflow mailing list > >core-workflow at python.org > >https://mail.python.org/mailman/listinfo/core-workflow > >This list is governed by the PSF Code of Conduct: > >https://www.python.org/psf/codeofconduct > > -- > Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From francismb at email.de Wed Jan 20 15:39:49 2016 From: francismb at email.de (francismb) Date: Wed, 20 Jan 2016 21:39:49 +0100 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: References: Message-ID: <569FF095.9060008@email.de> Hi Brett, > > Change sys._mercurial > ''''''''''''''''''''' > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` > attribute will need to be removed. An equivalent ``sys._git`` > attribute will be needed to take its place. > Would it make sense to change it to something more general like: sys._version_control_system or sys._vcs with maybe a new field for the type of vcs: 'mercurial', 'git' or 'the next one' :-) Regards, francis From guido at python.org Wed Jan 20 17:34:10 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 20 Jan 2016 14:34:10 -0800 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: <569FF095.9060008@email.de> References: <569FF095.9060008@email.de> Message-ID: For backwards compatibility sys._mercurial must probably stay. Maybe it could be None, so we could write if sys._mercurial: ... elif sys._git: ... else: # unknown VCS ... On Wed, Jan 20, 2016 at 12:39 PM, francismb wrote: > Hi Brett, > > > > Change sys._mercurial > > ''''''''''''''''''''' > > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` > > attribute will need to be removed. An equivalent ``sys._git`` > > attribute will be needed to take its place. > > > > Would it make sense to change it to something more general like: > > sys._version_control_system or > sys._vcs with maybe a new field for the type of vcs: 'mercurial', > 'git' or 'the next one' :-) > > > Regards, > francis > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Wed Jan 20 17:53:38 2016 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 21 Jan 2016 00:53:38 +0200 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: References: <569FF095.9060008@email.de> Message-ID: FWIW sys.subversion has been deprecated in 3.2.1 and removed in 3.3 [0]. In 3.2.1+ and 2.7 it returns ('CPython', '', ''). In theory sys._mercurial is private, so technically we don't have to go through the same deprecation process, but I see no harm in deprecating it in 3.6 and then removing it in 3.7+ (assuming we are going to remove it). Best Regards, Ezio Melotti [0]: https://docs.python.org/3.2/library/sys.html#sys.subversion On Thu, Jan 21, 2016 at 12:34 AM, Guido van Rossum wrote: > For backwards compatibility sys._mercurial must probably stay. Maybe it > could be None, so we could write > > if sys._mercurial: > ... > elif sys._git: > ... > else: > # unknown VCS ... > > On Wed, Jan 20, 2016 at 12:39 PM, francismb wrote: >> >> Hi Brett, >> > >> > Change sys._mercurial >> > ''''''''''''''''''''' >> > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` >> > attribute will need to be removed. An equivalent ``sys._git`` >> > attribute will be needed to take its place. >> > >> >> Would it make sense to change it to something more general like: >> >> sys._version_control_system or >> sys._vcs with maybe a new field for the type of vcs: 'mercurial', >> 'git' or 'the next one' :-) >> >> >> Regards, >> francis >> _______________________________________________ >> core-workflow mailing list >> core-workflow at python.org >> https://mail.python.org/mailman/listinfo/core-workflow >> This list is governed by the PSF Code of Conduct: >> https://www.python.org/psf/codeofconduct > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > core-workflow mailing list > core-workflow at python.org > https://mail.python.org/mailman/listinfo/core-workflow > This list is governed by the PSF Code of Conduct: > https://www.python.org/psf/codeofconduct From francismb at email.de Wed Jan 20 17:57:49 2016 From: francismb at email.de (francismb) Date: Wed, 20 Jan 2016 23:57:49 +0100 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: References: Message-ID: <56A010ED.4090802@email.de> Hi, > Repositories to Migrate > ======================= > While hg.python.org [#h.p.o]_ hosts many repositories, there are only > five key repositories that should move: > > 1. devinabox [#devinabox-repo]_ > 2. benchmarks [#benchmarks-repo]_ > 4. peps [#peps-repo]_ > 5. devguide [#devguide-repo]_ > 6. cpython [#cpython-repo]_ > On the web site I'm getting: """ While hg.python.org [1] hosts many repositories, there are only five key repositories that should move: devinabox [12] System Message: WARNING/2 ( pep-0512.txt , line 112) Enumerated list ends without a blank line; unexpected unindent. 2. benchmarks [11] 4. peps [13] 5. devguide [14] 6. cpython [15] """ Regards, francis From brett at python.org Thu Jan 21 12:21:56 2016 From: brett at python.org (Brett Cannon) Date: Thu, 21 Jan 2016 17:21:56 +0000 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: <569FF095.9060008@email.de> References: <569FF095.9060008@email.de> Message-ID: On Wed, 20 Jan 2016 at 12:40 francismb wrote: > Hi Brett, > > > > Change sys._mercurial > > ''''''''''''''''''''' > > Once Python is no longer kept in Mercurial, the ``sys._mercurial`` > > attribute will need to be removed. An equivalent ``sys._git`` > > attribute will be needed to take its place. > > > > Would it make sense to change it to something more general like: > > sys._version_control_system or > sys._vcs with maybe a new field for the type of vcs: 'mercurial', > 'git' or 'the next one' :-) > This is brought up every time we change the VCS. I'm not pushing one way or the other so I leave it up to others to argue whether we should change the approach taken by starting a new thread (but unless a consensus is reached I'm sticking with what we have done in the past). If I remember correctly, the original argument for not going generic is there is no guarantee future VCSs will have similar semantics that will fit into whatever tuple or dict structure we chose. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 21 23:24:30 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 22 Jan 2016 14:24:30 +1000 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: References: <569FF095.9060008@email.de> Message-ID: On 22 January 2016 at 03:21, Brett Cannon wrote: > If I remember correctly, the original argument for not going generic is > there is no guarantee future VCSs will have similar semantics that will fit > into whatever tuple or dict structure we chose. Yep, the name of the attribute conveys how to interpret it, while a generic name means you need some *other* data source to tell you "OK, up to version X.Y it's a subversion version, up to 3.5 it's a Mercurial hash, in 3.6+ it's a git hash..." With the attribute changing names, folks trying to use the VCS info at least get a really clear indicator when we change version control systems, even if they're not closely following upstream process changes. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Fri Jan 22 14:24:18 2016 From: brett at python.org (Brett Cannon) Date: Fri, 22 Jan 2016 19:24:18 +0000 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: References: <569FF095.9060008@email.de> Message-ID: On Thu, 21 Jan 2016 at 20:24 Nick Coghlan wrote: > On 22 January 2016 at 03:21, Brett Cannon wrote: > > If I remember correctly, the original argument for not going generic is > > there is no guarantee future VCSs will have similar semantics that will > fit > > into whatever tuple or dict structure we chose. > > Yep, the name of the attribute conveys how to interpret it, while a > generic name means you need some *other* data source to tell you "OK, > up to version X.Y it's a subversion version, up to 3.5 it's a > Mercurial hash, in 3.6+ it's a git hash..." > > With the attribute changing names, folks trying to use the VCS info at > least get a really clear indicator when we change version control > systems, even if they're not closely following upstream process > changes. > Actually, do people find the sys._mercurial attribute useful? I just want to double-check that this is worth continually going through this every few years when we change VCSs to swap in a new attribute. I will definitely update the PEP to set sys._mercurial to `('CPython', '', '')` (the attribute isn't even documented so there's no real deprecation to do). -------------- next part -------------- An HTML attachment was scrubbed... URL: From soltysh at gmail.com Fri Jan 22 17:44:45 2016 From: soltysh at gmail.com (Maciej Szulik) Date: Fri, 22 Jan 2016 23:44:45 +0100 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: References: Message-ID: On Wed, Jan 20, 2016 at 1:51 AM, Brett Cannon wrote: > Bot to handle pull request merging > '''''''''''''''''''''''''''''''''' > As stated in the section entitled > "`Document steps to commit a pull request`_", the desire is to > maintain a linear history for cpython. Unfortunately, > Github's [#github]_ web-based workflow does not support a linear > history. Because of this, a bot should be written to substitute for > GitHub's in-browser commit abilities. > > To start, the bot should accept commands to commit a pull request > against a list of branches. This allows for committing a pull request > that fixes a bug in multiple versions of Python. > > More advanced features such as a commit queue can come later. This > would linearly apply accepted pull requests and verify that the > commits did not interfere with each other by running the test suite > and backing out commits if the test run failed. To help facilitate > the speed of testing, all patches committed since the last test run > can be applied and run in a single test run as the optimistic > assumption is that the patches will work in tandem. > > Inspiration or basis of the bot could be taken from pre-existig bots > such as Homu [#homu]_ or Zuul [#zuul]_. > > > >From the experience on both OpenShift [1] and Kubernetes [2] I know there's a need to rerun tests quite frequently (flakes, errors, etc). One option is to close and re-open the issue to trigger the integrated CI services to re-run, which is cumbersome imho. Both of the aforementioned projects have testing bots. The one in orgin [1] is more sophisticated in that it allows core-devs running finer grained tests before actually merging the PR. The merge bot runs quite bit of the tests, but not all, possible examples include benchmarks. Just a food for thought ;) Maciej [1] https://github.com/openshift/origin/ [2] https://github.com/kubernetes/kubernetes -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jan 22 19:32:39 2016 From: brett at python.org (Brett Cannon) Date: Sat, 23 Jan 2016 00:32:39 +0000 Subject: [core-workflow] Updated draft of PEP 512 In-Reply-To: References: Message-ID: On Fri, 22 Jan 2016 at 14:44 Maciej Szulik wrote: > On Wed, Jan 20, 2016 at 1:51 AM, Brett Cannon wrote: > >> Bot to handle pull request merging >> '''''''''''''''''''''''''''''''''' >> As stated in the section entitled >> "`Document steps to commit a pull request`_", the desire is to >> maintain a linear history for cpython. Unfortunately, >> Github's [#github]_ web-based workflow does not support a linear >> history. Because of this, a bot should be written to substitute for >> GitHub's in-browser commit abilities. >> >> To start, the bot should accept commands to commit a pull request >> against a list of branches. This allows for committing a pull request >> that fixes a bug in multiple versions of Python. >> >> More advanced features such as a commit queue can come later. This >> would linearly apply accepted pull requests and verify that the >> commits did not interfere with each other by running the test suite >> and backing out commits if the test run failed. To help facilitate >> the speed of testing, all patches committed since the last test run >> can be applied and run in a single test run as the optimistic >> assumption is that the patches will work in tandem. >> >> Inspiration or basis of the bot could be taken from pre-existig bots >> such as Homu [#homu]_ or Zuul [#zuul]_. >> >> >> > From the experience on both OpenShift [1] and Kubernetes [2] I know > there's a need to > rerun tests quite frequently (flakes, errors, etc). One option is to close > and re-open the issue to > trigger the integrated CI services to re-run, which is cumbersome imho. > Both of the aforementioned > projects have testing bots. The one in orgin [1] is more sophisticated in > that it allows core-devs > running finer grained tests before actually merging the PR. The merge bot > runs quite bit > of the tests, but not all, possible examples include benchmarks. > Just a food for thought ;) > Thanks for the warning, Maciej. I have added a note that test re-runs should be considered in its design. Will go out with the next set of updates (hopefully tomorrow once I find out if anyone actually uses the sys._mercurial attribute). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jan 23 16:47:06 2016 From: brett at python.org (Brett Cannon) Date: Sat, 23 Jan 2016 21:47:06 +0000 Subject: [core-workflow] Starting work on PEP 512 Message-ID: Here is the latest version of the PEP. Since no one seems to be bringing up issues of missing steps or incorrect priorities, I think it's time to start work! The initial key todos relate to getting the ancillary PEPs moved over. I've taken on responsibility for writing the CLA bot (the pre-existing solutions don't seem to be maintained or are locked down to specific CLA signing solutions). The remaining items are: - Create a 'python-dev' team - Define commands to move a Mercurial repository to Git - Adding GitHub username support to bugs.python.org - How to update peps webpages from the future Git repo - How to update the devguide webpages from the future Git repo If anyone wants to step forward and help, then please do! I just ask you keep all of us up-to-date on what's going on. And if people want to work on some other task that's related to the cpython repo then that's fine as well, but do realize it might be quite a while before your work gets used (if at all, as things could potentially change). ---------- PEP: 512 Title: Migrating from hg.python.org to GitHub Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Discussions-To: core-workflow at python.org Status: Active Type: Process Content-Type: text/x-rst Created: 17-Jan-2015 Post-History: 17-Jan-2016, 19-Jan-2016, 23-Jan-2016 Abstract ======== This PEP outlines the steps required to migrate Python's development process from Mercurial [#hg]_ as hosted at hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting the minimum goals of this PEP should allow for the development process of Python to be as productive as it currently is, and meeting its extended goals should improve it. Rationale ========= In 2014, it became obvious that Python's custom development process was becoming a hindrance. As an example, for an external contributor to submit a fix for a bug that eventually was committed, the basic steps were: 1. Open an issue for the bug at bugs.python.org [#b.p.o]_. 2. Checkout out the CPython source code from hg.python.org [#h.p.o]_. 3. Make the fix. 4. Upload a patch. 5. Have a core developer review the patch using our fork of the Rietveld code review tool [#rietveld]_. 6. Download the patch to make sure it still applies cleanly. 7. Run the test suite manually. 8. Commit the change manually. 9. If the change was for a bugfix release, merge into the in-development branch. 10. Run the test suite manually again. 11. Commit the merge. 12. Push the changes. This is a very heavy, manual process for core developers. Even in the simple case, you could only possibly skip the code review step, as you would still need to build the documentation. This led to patches languishing on the issue tracker due to core developers not being able to work through the backlog fast enough to keep up with submissions. In turn, that led to a side-effect issue of discouraging outside contribution due to frustration from lack of attention, which is dangerous problem for an open source project as it runs counter to having a viable future for the project. Simply accepting patches uploaded to bugs.python.org [#b.p.o]_ is potentially simple for an external contributor, it is as slow and burdensome as it gets for a core developer to work with. Hence the decision was made in late 2014 that a move to a new development process was needed. A request for PEPs proposing new workflows was made, in the end leading to two: PEP 481 and PEP 507 proposing GitHub [#github]_ and GitLab [#gitlab]_, respectively. The year 2015 was spent off-and-on working on those proposals and trying to tease out details of what made them different from each other on the core-workflow mailing list [#core-workflow]_. PyCon US 2015 also showed that the community was a bit frustrated with our process due to both cognitive overhead for new contributors and how long it was taking for core developers to look at a patch (see the end of Guido van Rossum's keynote at PyCon US 2015 [#guido-keynote]_ as an example of the frustration). On January 1, 2016, the decision was made by Brett Cannon to move the development process to GitHub. The key reasons for choosing GitHub were [#reasons]_: * Maintaining custom infrastructure has been a burden on volunteers (e.g., a custom fork of Rietveld [#rietveld]_ that is not being maintained is currently being used). * The custom workflow is very time-consuming for core developers (not enough automated tooling built to help support it). * The custom workflow is a hindrance to external contributors (acts as a barrier of entry due to time required to ramp up on development process). * There is no feature differentiating GitLab from GitHub beyond GitLab being open source. * Familiarity with GitHub is far higher amongst core developers and external contributors than with GitLab. * Our BDFL prefers GitHub (who would be the first person to tell you that his opinion shouldn't matter, but the person making the decision felt it was important that the BDFL feel comfortable with the workflow of his own programming language to encourage his continued participation). There's even already an unofficial image to use to represent the migration to GitHub [#pythocat]_. The overarching goal of this migration is to improve the development process to the extent that a core developer can go from external contribution submission through all the steps leading to committing said contribution all from within a browser on a tablet with WiFi using *some* development process (this does not inherently mean GitHub's default workflow). All of this will be done in such a way that if an external contributor chooses not to use GitHub then they will continue to have that option. Repositories to Migrate ======================= While hg.python.org [#h.p.o]_ hosts many repositories, there are only five key repositories that should move: 1. devinabox [#devinabox-repo]_ 2. benchmarks [#benchmarks-repo]_ 3. peps [#peps-repo]_ 4. devguide [#devguide-repo]_ 5. cpython [#cpython-repo]_ The devinabox and benchmarks repositories are code-only. The peps and devguide repositories involve the generation of webpages. And the cpython repository has special requirements for integration with bugs.python.org [#b.p.o]_. Migration Plan ============== The migration plan is separated into sections based on what is required to migrate the repositories listed in the `Repositories to Migrate`_ section. Completion of requirements outlined in each section should unblock the migration of the related repositories. The sections are expected to be completed in order, but not necessarily the requirements within a section. Requirements for Code-Only Repositories --------------------------------------- Completion of the requirements in this section will allow the devinabox and benchmarks repositories to move to GitHub. While devinabox has a sufficiently descriptive name, the benchmarks repository does not; therefore, it will be named "python-benchmark-suite". Create a 'python-dev' team '''''''''''''''''''''''''' To manage permissions, a 'python-dev' team will be created as part of the python organization [#github-python-org]_. Any repository that is moved will have the 'python-dev' team added to it with write permissions [#github-org-perms]_. Anyone who previously had rights to manage SSH keys on hg.python.org will become a team maintainer for the 'python-dev' team. Define commands to move a Mercurial repository to Git ''''''''''''''''''''''''''''''''''''''''''''''''''''' Since moving to GitHub also entails moving to Git [#git]_, we must decide what tools and commands we will run to translate a Mercurial repository to Git. The exact tools and steps to use are an open issue; see `Tools and commands to move from Mercurial to Git`_. CLA enforcement ''''''''''''''' A key part of any open source project is making sure that its source code can be properly licensed. This requires making sure all people making contributions have signed a contributor license agreement (CLA) [#cla]_. Up until now, enforcement of CLA signing of contributed code has been enforced by core developers checking whether someone had an ``*`` by their username on bugs.python.org [#b.p.o]_. With this migration, the plan is to start off with automated checking and enforcement of contributors signing the CLA. Adding GitHub username support to bugs.python.org +++++++++++++++++++++++++++++++++++++++++++++++++ To keep tracking of CLA signing under the direct control of the PSF, tracking who has signed the PSF CLA will be continued by marking that fact as part of someone's bugs.python.org user profile. What this means is that an association will be needed between a person's bugs.python.org [#b.p.o]_ account and their GitHub account, which will be done through a new field in a user's profile. This does implicitly require that contributors will need both a GitHub [#github]_ and bugs.python.org account in order to sign the CLA and contribute through GitHub. A bot to enforce CLA signing ++++++++++++++++++++++++++++ With an association between someone's GitHub account and their bugs.python.org [#b.p.o]_ account, which has the data as to whether someone has signed the CLA, a bot can monitor pull requests on GitHub and denote whether the contributor has signed the CLA. If the user has signed the CLA, the bot will add a positive label to the issue to denote the pull request has no CLA issues (e.g., a green label stating, "CLA: ?"). If the contributor has not signed a CLA, a negative label will be added to the pull request will be blocked using GitHub's status API (e.g., a red label stating, "CLA: ?"). If a contributor lacks a bugs.python.org account, that will lead to another label (e.g., "CLA: ? (no account)"). Using a label for both positive and negative cases provides a fallback notification if the bot happens to fail, preventing potential false-positives or false-negatives. It also allows for an easy way to trigger the bot again by simply removing a CLA-related label. If no pre-existing, maintained bot exists that fits our needs, one will be written from scratch. It will be hosted on Heroku [#heroku]_ and written to target Python 3.5 to act as a showcase for asynchronous programming. The bot's actual name is an open issue: `Naming the bots`_ Requirements for Web-Related Repositories ----------------------------------------- Due to their use for generating webpages, the devguide [#devguide-repo]_ and peps [#peps-repo]_ repositories need their respective processes updated to pull from their new Git repositories. The devguide repository might also need to be named ``python-devguide`` to make sure the repository is not ambiguous when viewed in isolation from the python organization [#github-python-org]_. Requirements for the cpython Repository --------------------------------------- Obviously the most active and important repository currently hosted at hg.python.org [#h.p.o]_ is the cpython repository [#cpython-repo]_. Because of its importance and high- frequency use, it requires more tooling before being moved to GitHub compared to the other repositories mentioned in this PEP. Document steps to commit a pull request ''''''''''''''''''''''''''''''''''''''' During the process of choosing a new development workflow, it was decided that a linear history is desired. People preferred having a single commit representing a single change instead of having a set of unrelated commits lead to a merge commit that represented a single change. This means that the convenient "Merge" button in GitHub pull requests is undesirable, as it creates a merge commit along with all of the contributor's individual commits (this does not affect the other repositories where the desire for a linear history doesn't exist). Luckily, Git [#git]_ does not require GitHub's workflow and so one can be chosen which gives us a linear history by using Git's CLI. The expectation is that all pull requests will be fast-forwarded and rebased before being pushed to the master repository. This should give proper attribution to the pull request author in the Git history. This does have the consequence of losing some GitHub features such as automatic closing of pull requests, link generation, etc. A second set of recommended commands will also be written for committing a contribution from a patch file uploaded to bugs.python.org [#b.p.o]_. This will obviously help keep the linear history, but it will need to be made to have attribution to the patch author. The exact sequence of commands that will be given as guidelines to core developers is an open issue: `Git CLI commands for committing a pull request to cpython`_. Handling Misc/NEWS '''''''''''''''''' Traditionally the ``Misc/NEWS`` file [#news-file]_ has been problematic for changes which spanned Python releases. Often times there will be merge conflicts when committing a change between e.g., 3.5 and 3.6 only in the ``Misc/NEWS`` file. It's so common, in fact, that the example instructions in the devguide explicitly mention how to resolve conflicts in the ``Misc/NEWS`` file [#devguide-merge-across-branches]_. As part of our tool modernization, working with the ``Misc/NEWS`` file will be simplified. There are currently two competing approaches to solving the ``Misc/NEWS`` problem which are discussed in an open issue: `How to handle the Misc/NEWS file`_. Handling Misc/ACKS '''''''''''''''''' Traditionally the ``Misc/ACKS`` file [#acks-file]_ has been managed by hand. But thanks to Git supporting an ``author`` value as well as a ``committer`` value per commit, authorship of a commit can be part of the history of the code itself. As such, manual management of ``Misc/ACKS`` will become optional. A script will be written that will collect all author and committer names and merge them into ``Misc/ACKS`` with all of the names listed prior to the move to Git. Running this script will become part of the release process. Linking pull requests to issues ''''''''''''''''''''''''''''''' Historically, external contributions were attached to an issue on bugs.python.org [#b.p.o]_ thanks to the fact that all external contributions were uploaded as a file. For changes committed by a core developer who committed a change directly, the specifying of an issue number in the commit message of the format ``Issue #`` at the start of the message led to a comment being posted to the issue linking to the commit. Linking a pull request to an issue ++++++++++++++++++++++++++++++++++ An association between a pull request and an issue is needed to track when a fix has been proposed. The association needs to be many-to-one as there can take multiple pull requests to solve a single issue (technically it should be a many-to-many association for when a single fix solves multiple issues, but this is fairly rare and issues can be merged into one using the ``Superseder`` field on the issue tracker). Association between a pull request and an issue will be done based on detecting the regular expression``[Ii]ssue #(?P\d+)``. If this is specified in either the title or in the body of a message on a pull request then connection will be made on bugs.python.org [#b.p.o]_. A label will also be added to the pull request to signify that the connection was made successfully. This could lead to incorrect associations if the wrong issue or referencing another issue was done, but these are rare occasions. Notify the issue if the pull request is committed +++++++++++++++++++++++++++++++++++++++++++++++++ Once a pull request is closed (merged or not), the issue should be updated to reflect this fact. Update linking service for mapping commit IDs to URLs ''''''''''''''''''''''''''''''''''''''''''''''''''''' Currently you can use https://hg.python.org/lookup/ with a revision ID from either the Subversion or Mercurial copies of the cpython repo [#cpython-repo]_ to get redirected to the URL for that revision in the Mercurial repository. The URL rewriter will need to be updated to redirect to the Git repository and to support the new revision IDs created for the Git repository. Create https://git.python.org ''''''''''''''''''''''''''''' Just as hg.python.org [#h.p.o]_ currently points to the Mercurial repository for Python, git.python.org should do the equivalent for the Git repository. Backup of pull request data ''''''''''''''''''''''''''' Since GitHub [#github]_ is going to be used for code hosting and code review, those two things need to be backed up. In the case of code hosting, the backup is implicit as all non-shallow Git [#git]_ clones contain the full history of the repository, hence there will be many backups of the repository. The code review history does not have the same implicit backup mechanism as the repository itself. That means a daily backup of code review history should be done so that it is not lost in case of any issues with GitHub. It also helps guarantee that a migration from GitHub to some other code review system is feasible were GitHub to disappear overnight. Deprecate sys._mercurial '''''''''''''''''''''''' Once Python is no longer kept in Mercurial, the ``sys._mercurial`` attribute will need to be changed to return ``('CPython', '', '')``. An equivalent ``sys._git`` attribute will be added which fulfills the same use-cases. Update the devguide ''''''''''''''''''' The devguide will need to be updated with details of the new workflow. Mostly likely work will take place in a separate branch until the migration actually occurs. Update PEP 101 '''''''''''''' The release process will need to be updated as necessary. Optional, Planned Features -------------------------- Once the cpython repository [#cpython-repo]_ is migrated, all repositories will have been moved to GitHub [#github]_ and the development process should be on equal footing as before. But a key reason for this migration is to improve the development process, making it better than it has ever been. This section outlines some plans on how to improve things. It should be mentioned that overall feature planning for bugs.python.org [#b.p.o]_ -- which includes plans independent of this migration -- are tracked on their own wiki page [#tracker-plans]_. Bot to handle pull request merging '''''''''''''''''''''''''''''''''' As stated in the section entitled "`Document steps to commit a pull request`_", the desire is to maintain a linear history for cpython. Unfortunately, Github's [#github]_ web-based workflow does not support a linear history. Because of this, a bot should be written to substitute for GitHub's in-browser commit abilities. To start, the bot should accept commands to commit a pull request against a list of branches. This allows for committing a pull request that fixes a bug in multiple versions of Python. More advanced features such as a commit queue can come later. This would linearly apply accepted pull requests and verify that the commits did not interfere with each other by running the test suite and backing out commits if the test run failed. To help facilitate the speed of testing, all patches committed since the last test run can be applied and run in a single test run as the optimistic assumption is that the patches will work in tandem. Some mechanism to re-run the tests in case of test flakiness will be needed, whether it is from removing a "test failed" label, web interface for core developers to trigger another testing event, etc. Inspiration or basis of the bot could be taken from pre-existig bots such as Homu [#homu]_ or Zuul [#zuul]_. The name given to this bot in order to give it commands is an open issue: `Naming the bots`_. Continuous integration per pull request ''''''''''''''''''''''''''''''''''''''' To help speed up pull request approvals, continuous integration testing should be used. This helps mitigate the need for a core developer to download a patch simply to run the test suite against the patch. Which free CI service to use is an open issue: `Choosing a CI service`_. Test coverage report '''''''''''''''''''' Getting an up-to-date test coverage report for Python's standard library would be extremely beneficial as generating such a report can take quite a while to produce. There are a couple pre-existing services that provide free test coverage for open source projects. Which option is best is an open issue: `Choosing a test coverage service`_. Notifying issues of pull request comments ''''''''''''''''''''''''''''''''''''''''' The current development process does not include notifying an issue on bugs.python.org [#b.p.o]_ when a review comment is left on Rietveld [#rietveld]_. It would be nice to fix this so that people can subscribe only to comments at bugs.python.org and not GitHub [#github]_ and yet still know when something occurs on GitHub in terms of review comments on relevant pull requests. Current thinking is to post a comment to bugs.python.org to the relevant issue when at least one review comment has been made over a certain period of time (e.g., 15 or 30 minutes). This keeps the email volume down for those that receive both GitHub and bugs.python.org email notifications while still making sure that those only following bugs.python.org know when there might be a review comment to address. Allow bugs.python.org to use GitHub as a login provider ''''''''''''''''''''''''''''''''''''''''''''''''''''''' As of right now, bugs.python.org [#b.p.o]_ allows people to log in using Google, Launchpad, or OpenID credentials. It would be good to expand this to GitHub credentials. Web hooks for re-generating web content ''''''''''''''''''''''''''''''''''''''' The content at https://docs.python.org/, https://docs.python.org/devguide, and https://www.python.org/dev/peps/ are all derived from files kept in one of the repositories to be moved as part of this migration. As such, it would be nice to set up appropriate webhooks to trigger rebuilding the appropriate web content when the files they are based on change instead of having to wait for, e.g., a cronjob to trigger. Link web content back to files that it is generated from '''''''''''''''''''''''''''''''''''''''''''''''''''''''' It would be helpful for people who find issues with any of the documentation that is generated from a file to have a link on each page which points back to the file on GitHub [#github]_ that stores the content of the page. That would allow for quick pull requests to fix simple things such as spelling mistakes. Splitting out parts of the documentation into their own repositories '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' While certain parts of the documentation at https://docs.python.org change with the code, other parts are fairly static and are not tightly bound to the CPython code itself. The following sections of the documentation fit this category of slow-changing, loosely-coupled: * `Tutorial `__ * `Python Setup and Usage `__ * `HOWTOs `__ * `Installing Python Modules < https://docs.python.org/3/installing/index.html>`__ * `Distributing Python Modules < https://docs.python.org/3/distributing/index.html>`__ * `Extending and Embedding `__ * `FAQs `__ These parts of the documentation could be broken out into their own repositories to simplify their maintenance and to expand who has commit rights to them to ease in their maintenance. It has also been suggested to split out the `What's New `__ documents. That would require deciding whether a workflow could be developed where it would be difficult to forget to update What's New (potentially through a label added to PRs, like "What's New needed"). Backup of Git repositories '''''''''''''''''''''''''' While not necessary, it would be good to have official backups of the various Git repositories for disaster protection. It will be up to the PSF infrastructure committee to decide if this is worthwhile or unnecessary. Status ====== Requirements for migrating the devinabox [#devinabox-repo]_ and benchmarks [#benchmarks-repo]_ repositories: * Not started - `Create a 'python-dev' team`_ - `Define commands to move a Mercurial repository to Git`_ - `Adding GitHub username support to bugs.python.org`_ * In progress - `A bot to enforce CLA signing`_: https://github.com/brettcannon/knights-who-say-ni (Brett Cannon) * Completed - None Repositories whose build steps need updating: * Not started - peps [#peps-repo]_ - devguide [#devguide-repo]_ * In progress - None * Completed - None Requirements to move over the cpython repo [#cpython-repo]_: * Not started - `Document steps to commit a pull request`_ - `Handling Misc/NEWS`_ - `Handling Misc/ACKS`_ - `Linking a pull request to an issue`_ - `Notify the issue if the pull request is committed`_ - `Update linking service for mapping commit IDs to URLs`_ - `Create https://git.python.org`_ - `Backup of pull request data`_ - `Deprecate sys._mercurial`_ - `Update the devguide`_ - `Update PEP 101`_ * In progress - None * Completed - None Optional features: * Not started - `Bot to handle pull request merging`_ - `Continuous integration per pull request`_ - `Test coverage report`_ - `Notifying issues of pull request comments`_ - `Allow bugs.python.org to use GitHub as a login provider`_ - `Web hooks for re-generating web content`_ - `Link web content back to files that it is generated from`_ - `Splitting out parts of the documentation into their own repositories`_ - `Backup of Git repositories`_ * In progress - None * Completed - None Open Issues =========== For this PEP, open issues are ones where a decision needs to be made to how to approach or solve a problem. Open issues do not entail coordination issues such as who is going to write a certain bit of code. The fate of hg.python.org ------------------------- With the code repositories moving over to Git [#git]_, there is no technical need to keep hg.python.org [#h.p.o]_ running. Having said that, some in the community would like to have it stay functioning as a Mercurial [#hg]_ mirror of the Git repositories. Others have said that they still want a mirror, but one using Git. As maintaining hg.python.org is not necessary, it will be up to the PSF infrastructure committee to decide if they want to spend the time and resources to keep it running. They may also choose whether they want to host a Git mirror on PSF infrastructure. Depending on the decision reached, other ancillary repositories will either be forced to migration or they can choose to simply stay on hg.python.org. Tools and commands to move from Mercurial to Git ------------------------------------------------ A decision needs to be made on exactly what tooling and what commands involving those tools will be used to convert a Mercurial repository to Git. Currently a suggestion has been made to use https://github.com/frej/fast-export. Another suggestion is to use https://github.com/felipec/git-remote-hg. Finally, http://hg-git.github.io/ has been suggested. Git CLI commands for committing a pull request to cpython --------------------------------------------------------- Because Git [#git]_ may be a new version control system for core developers, the commands people are expected to run will need to be written down. These commands also need to keep a linear history while giving proper attribution to the pull request author. Another set of commands will also be necessary for when working with a patch file uploaded to bugs.python.org [#b.p.o]_. Here the linear history will be kept implicitly, but it will need to make sure to keep/add attribution. How to handle the Misc/NEWS file -------------------------------- There are three competing approaches to handling ``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on bugs.python.org [#b.p.o]_. This would mean an issue that is marked as "resolved" could not be closed until a news entry is added in the "news" field in the issue tracker. The benefit of tying the news entry to the issue is it makes sure that all changes worthy of a news entry have an accompanying issue. It also makes classifying a news entry automatic thanks to the Component field of the issue. The Versions field of the issue also ties the news entry to which Python releases were affected. A script would be written to query bugs.python.org for relevant new entries for a release and to produce the output needed to be checked into the code repository. This approach is agnostic to whether a commit was done by CLI or bot. A competing approach is to use an individual file per news entry, containing the text for the entry. In this scenario each feature release would have its own directory for news entries and a separate file would be created in that directory that was either named after the issue it closed or a timestamp value (which prevents collisions). Merges across branches would have no issue as the news entry file would still be uniquely named and in the directory of the latest version that contained the fix. A script would collect all news entry files no matter what directory they reside in and create an appropriate news file (the release directory can be ignored as the mere fact that the file exists is enough to represent that the entry belongs to the release). Classification can either be done by keyword in the new entry file itself or by using subdirectories representing each news entry classification in each release directory (or classification of news entries could be dropped since critical information is captured by the "What's New" documents which are organized). The benefit of this approach is that it keeps the changes with the code that was actually changed. It also ties the message to being part of the commit which introduced the change. For a commit made through the CLI, a script will be provided to help generate the file. In a bot-driven scenario, the merge bot will have a way to specify a specific news entry and create the file as part of its flattened commit (while most likely also supporting using the first line of the commit message if no specific news entry was specified). Code for this approach has been written previously for the Mercurial workflow at http://bugs.python.org/issue18967. There is also tools from the community like https://pypi.python.org/pypi/towncrier and https://github.com/twisted/newsbuilder . A yet third option is a merge script to handle the conflicts. This approach allows for keeping the NEWS file as a single file. It does run the risk, though, of failure and thus blocking a commit until it can be manually resolved. Naming the bots --------------- As naming things can lead to bikeshedding of epic proportions, Brett Cannon will choose the final name of the various bots (the name of the project for the bots themselves can be anything, this is purely for the name used in giving commands to the bot or the account name). The names will come from Monty Python, which is only fitting since Python is named after the comedy troupe. They will most likely come from 'Monty Python and the Holy Grail' [#holy-grail]_ (which happens to be how Brett was introduced to Monty Python). Current ideas on the name include: "Black Knight" sketch [#black-knight-sketch]_: * black-knight * none-shall-pass * just-a-flesh-wound "Bridge of Death" sketch [#bridge-of-death-sketch]_: * bridge-keeper * man-from-scene-24 * five-questions * what-is-your-quest * blue-no-green * air-speed-velocity * your-favourite-colour (and that specific spelling; Monty Python is British, after all) "Killer rabbit" sketch [#killer-rabbit-sketch]_: * killer-rabbit * holy-hand-grenade * 5-is-right-out "French Taunter" sketch [#french-taunter-sketch]_: * elderberries * kanigget "Constitutional Peasants" sketch [#constitutional-peasants-sketch]_: * dennis * from-the-masses "Knights Who Say 'Ni'" sketch [#ni-sketch]_: * shubbery * ni * knights-who-say-ni >From "Monty Python and the Holy Grail" in general: * brave-sir-robin Choosing a CI service --------------------- There are various CI services that provide free support for open source projects hosted on GitHub [#github]_. Two such examples are Travis [#travis]_ and Codeship [#codeship]_. Whatever solution is chosen will need to not time out in the time it takes to execute Python's test suite. It should optimally provide access to multiple C compilers for more thorough testing. Network access is also beneficial. The current CI service for Python is Pypatcher [#pypatcher]_. A request can be made in IRC to try a patch from bugs.python.org [#b.p.o]_. The results can be viewed at https://ci.centos.org/job/cPython-build-patch/ . Choosing a test coverage service -------------------------------- Getting basic test coverage of Python's standard library can be created simply by using coverage.py [#coverage]_. Getting thorough test coverage is actually quite tricky, with the details outlined in the devinabox's README [#devinabox-repo]_. It would be best if a service could be found that would allow for thorough test coverage, but it might not be feasible. Free test coverage services include Coveralls [#coveralls]_ and Codecov [#codecov]_. Rejected Ideas ============== Separate Python 2 and Python 3 repositories ------------------------------------------- It was discussed whether separate repositories for Python 2 and Python 3 were desired. The thinking was that this would shrink the overall repository size which benefits people with slow Internet connections or small bandwidth caps. In the end it was decided that it was easier logistically to simply keep all of CPython's history in a single repository. Commit multi-release changes in bugfix branch first --------------------------------------------------- As the current development process has changes committed in the oldest branch first and then merged up to the default branch, the question came up as to whether this workflow should be perpetuated. In the end it was decided that committing in the newest branch and then cherry-picking changes into older branches would work best as most people will instinctively work off the newest branch and it is a more common workflow when using Git [#git]_. Cherry-picking is also more bot-friendly for an in-browser workflow. In the merge-up scenario, if you were to request a bot to do a merge and it failed, then you would have to make sure to immediately solve the merge conflicts if you still allowed the main commit, else you would need to postpone the entire commit until all merges could be handled. With a cherry-picking workflow, the main commit could proceed while postponing the merge-failing cherry-picks. This allows for possibly distributing the work of managing conflicting merges. Deriving ``Misc/NEWS`` from the commit logs ------------------------------------------- As part of the discussion surrounding `Handling Misc/NEWS`_, the suggestion has come up of deriving the file from the commit logs itself. In this scenario, the first line of a commit message would be taken to represent the news entry for the change. Some heuristic to tie in whether a change warranted a news entry would be used, e.g., whether an issue number is listed. This idea has been rejected due to some core developers preferring to write a news entry separate from the commit message. The argument is the first line of a commit message compared to that of a news entry have different requirements in terms of brevity, what should be said, etc. References ========== .. [#h.p.o] https://hg.python.org .. [#GitHub] GitHub (https://github.com) .. [#hg] Mercurial (https://www.mercurial-scm.org/) .. [#git] Git (https://git-scm.com/) .. [#b.p.o] https://bugs.python.org .. [#rietveld] Rietveld (https://github.com/rietveld-codereview/rietveld) .. [#gitlab] GitLab (https://about.gitlab.com/) .. [#core-workflow] core-workflow mailing list ( https://mail.python.org/mailman/listinfo/core-workflow) .. [#guido-keynote] Guido van Rossum's keynote at PyCon US ( https://www.youtube.com/watch?v=G-uKNd5TSBw) .. [#reasons] Email to core-workflow outlining reasons why GitHub was selected (https://mail.python.org/pipermail/core-workflow/2016-January/000345.html ) .. [#benchmarks-repo] Mercurial repository for the Unified Benchmark Suite (https://hg.python.org/benchmarks/) .. [#devinabox-repo] Mercurial repository for devinabox ( https://hg.python.org/devinabox/) .. [#peps-repo] Mercurial repository of the Python Enhancement Proposals ( https://hg.python.org/peps/) .. [#devguide-repo] Mercurial repository for the Python Developer's Guide ( https://hg.python.org/devguide/) .. [#cpython-repo] Mercurial repository for CPython ( https://hg.python.org/cpython/) .. [#github-python-org] Python organization on GitHub ( https://github.com/python) .. [#github-org-perms] GitHub repository permission levels ( https://help.github.com/enterprise/2.4/user/articles/repository-permission-levels-for-an-organization/ ) .. [#cla] Python Software Foundation Contributor Agreement ( https://www.python.org/psf/contrib/contrib-form/) .. [#news-file] ``Misc/NEWS`` ( https://hg.python.org/cpython/file/default/Misc/NEWS) .. [#acks-file] ``Misc/ACKS`` ( https://hg.python.org/cpython/file/default/Misc/ACKS) .. [#devguide-merge-across-branches] Devguide instructions on how to merge across branches ( https://docs.python.org/devguide/committing.html#merging-between-different-branches-within-the-same-major-version ) .. [#pythocat] Pythocat (https://octodex.github.com/pythocat/) .. [#tracker-plans] Wiki page for bugs.python.org feature development (https://wiki.python.org/moin/TrackerDevelopmentPlanning) .. [#black-knight-sketch] The "Black Knight" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=dhRUe-gz690) .. [#bridge-of-death-sketch] The "Bridge of Death" sketch from "Monty Python and the Holy Grail" (https://www.youtube.com/watch?v=cV0tCphFMr8) .. [#holy-grail] "Monty Python and the Holy Grail" sketches (https://www.youtube.com/playlist?list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp ) .. [#killer-rabbit-sketch] "Killer rabbit" sketch from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=Nvs5pqf-DMA&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=11 ) .. [#french-taunter-sketch] "French Taunter" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=A8yjNbcKkNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=13 ) .. [#constitutional-peasants-sketch] "Constitutional Peasants" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=JvKIWjnEPNY&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=14 ) .. [#ni-sketch] "Knights Who Say Ni" from "Monty Python and the Holy Grail" ( https://www.youtube.com/watch?v=zIV4poUZAQo&list=PL-Qryc-SVnnu1MvN3r94Y9atpaRuIoGmp&index=15 ) .. [#homu] Homu (http://homu.io/) .. [#zuul] Zuul (http://docs.openstack.org/infra/zuul/) .. [#travis] Travis (https://travis-ci.org/) .. [#codeship] Codeship (https://codeship.com/) .. [#coverage] coverage.py (https://pypi.python.org/pypi/coverage) .. [#coveralls] Coveralls (https://coveralls.io/) .. [#codecov] Codecov (https://codecov.io/) .. [#pypatcher] Pypatcher (https://github.com/kushaldas/pypatcher) .. [#heroku] Heroku (https://www.heroku.com/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: