From jeremy.kloth at gmail.com Thu Nov 3 14:40:43 2016 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Thu, 3 Nov 2016 12:40:43 -0600 Subject: [Python-buildbots] Windows 2.7 Builders Message-ID: Hello, I was recently cleaning up my buildbot, clearing space, removing unused software, ..., and decided to remove MSVC 2010 as no supported Python uses that compiler. Well, to my surprise, actually I forgot, you do need MSVC 2010 *AND* 2008 to build 2.7 by default. It has been since reinstalled and the buildbot is chugging along happily. With that, do we have any buildbots that use *just* 2008? Or does the 2.7 compiler package from Microsoft contain the needed bits that are provided by installing Visual Studio 2010? I think my buildslave has enough resources to support another builder for that configuration (VC9 only) if that is something we are interested in doing. Thanks -- Jeremy Kloth From zachary.ware+pydev at gmail.com Thu Nov 3 15:35:02 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 3 Nov 2016 14:35:02 -0500 Subject: [Python-buildbots] Windows 2.7 Builders In-Reply-To: References: Message-ID: Hi Jeremy, On Thu, Nov 3, 2016 at 1:40 PM, Jeremy Kloth wrote: > Hello, > > I was recently cleaning up my buildbot, clearing space, removing > unused software, ..., and decided to remove MSVC 2010 as no supported > Python uses that compiler. Well, to my surprise, actually I forgot, > you do need MSVC 2010 *AND* 2008 to build 2.7 by default. It has been > since reinstalled and the buildbot is chugging along happily. > > With that, do we have any buildbots that use *just* 2008? Or does the > 2.7 compiler package from Microsoft contain the needed bits that are > provided by installing Visual Studio 2010? Unfortunately, the bits needed from VS 2010 are entirely separate from the bits provided by the 2.7 compiler package; the compiler package is more closely related to VS 2008, but still missing some important pieces. > I think my buildslave has enough resources to support another builder > for that configuration (VC9 only) if that is something we are > interested in doing. I am in fact interested in doing so; there have been a couple of issues raised recently about the PC/VS9.0 files being left behind. I may also add a VS2008-only builder to the XP bot, since it's only running 2.7 anymore anyways. -- Zach From db3l.net at gmail.com Thu Nov 3 15:38:15 2016 From: db3l.net at gmail.com (David Bolen) Date: Thu, 3 Nov 2016 15:38:15 -0400 Subject: [Python-buildbots] Windows 2.7 Builders In-Reply-To: References: Message-ID: Interesting - I probably would have assumed removing 2010 was safe for 2.7 as well. However, all of my 2.7 builders (XP/7/8/10) currently have both 2008 and 2010 so any dependency on 2010 would go unnoticed. Assuming the 2010 dependency is unintentional, or we specifically want to test without it, the XP slave no longer builds anything but 2.7, so I could actually uninstall 2010 from that machine without affecting any other builders, or requiring any special configuration to avoid a 2010 installation. -- David On Thu, Nov 3, 2016 at 2:40 PM, Jeremy Kloth wrote: > > Hello, > > I was recently cleaning up my buildbot, clearing space, removing > unused software, ..., and decided to remove MSVC 2010 as no supported > Python uses that compiler. Well, to my surprise, actually I forgot, > you do need MSVC 2010 *AND* 2008 to build 2.7 by default. It has been > since reinstalled and the buildbot is chugging along happily. > > With that, do we have any buildbots that use *just* 2008? Or does the > 2.7 compiler package from Microsoft contain the needed bits that are > provided by installing Visual Studio 2010? > > I think my buildslave has enough resources to support another builder > for that configuration (VC9 only) if that is something we are > interested in doing. > > Thanks > > -- > Jeremy Kloth > _______________________________________________ > Python-Buildbots mailing list > Python-Buildbots at python.org > https://mail.python.org/mailman/listinfo/python-buildbots From zachary.ware+pydev at gmail.com Thu Nov 3 15:55:42 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 3 Nov 2016 14:55:42 -0500 Subject: [Python-buildbots] Windows 2.7 Builders In-Reply-To: References: Message-ID: Hi David, On Thu, Nov 3, 2016 at 2:38 PM, David Bolen wrote: > Interesting - I probably would have assumed removing 2010 was safe for > 2.7 as well. However, all of my 2.7 builders (XP/7/8/10) currently > have both 2008 and 2010 so any dependency on 2010 would go unnoticed. > > Assuming the 2010 dependency is unintentional, or we specifically want > to test without it, the XP slave no longer builds anything but 2.7, so > I could actually uninstall 2010 from that machine without affecting > any other builders, or requiring any special configuration to avoid a > 2010 installation. The dependency on both 2008 and 2010 is intentional for the build files in PCbuild/. The reason is that I backported the new build files from 3.5 to allow for easier ICC support (and a few other minor benefits), but to make that work we need the compiler toolchain from 2008 and the v90 PlatformToolset files from 2010 to use with an MSBuild from 2010 or later. PC/VS9.0 is completely independent, though, and has no reliance on VS 2010. I'd like to just add a builder to your XP bot that builds in PC/VS9.0 in addition to the regular builder in PCbuild. That will allow us to keep track of whether a change breaks one build or both. Once I have a VS9-only builder set up, if you want to remove VS2010 from any of your other bots, let me know and I'll switch its 2.7 builder to use VS9-only. -- Zach From db3l.net at gmail.com Thu Nov 3 16:16:04 2016 From: db3l.net at gmail.com (David Bolen) Date: Thu, 3 Nov 2016 16:16:04 -0400 Subject: [Python-buildbots] Windows 2.7 Builders In-Reply-To: References: Message-ID: Zachary, > I'd like to just add a builder to your XP bot that builds in PC/VS9.0 > in addition to the regular builder in PCbuild. That will allow us to > keep track of whether a change breaks one build or both. Once I have > a VS9-only builder set up, if you want to remove VS2010 from any of > your other bots, let me know and I'll switch its 2.7 builder to use > VS9-only. The additional builder sounds fine. I don't have any particular need to remove 2010 anywhere, but wasn't sure if perhaps by having it still installed we were glossing over something on the buildslaves that a third party might run into, which doesn't seem to be the case. In terms of switching any other builders over, I suppose the intent should be to use whatever scripts are used for the final python.org builds, so we test what will be distributed. I realize the actual compiler in use has far more ramifications than the build process, but might as well stay as aligned as possible. I'm guessing that's PCBuild, so no change to the existing builders, and the buildslaves keep the 2010 dependency for 2.7. But for third party independent builds, it appears like the 2.7.12 documentation (https://docs.python.org/2/using/windows.html) still lists PCBuild and 2008. So perhaps that should be updated to PC/VS9.0 similar to the earlier versions? At which point, keeping at least one builder using PC/VS9.0 definitely seems like a good idea. -- David From db3l.net at gmail.com Thu Nov 3 16:23:32 2016 From: db3l.net at gmail.com (David Bolen) Date: Thu, 3 Nov 2016 16:23:32 -0400 Subject: [Python-buildbots] Windows 2.7 Builders In-Reply-To: References: Message-ID: BTW, just to follow up on my last message: > But for third party independent builds, it appears like the 2.7.12 > documentation (https://docs.python.org/2/using/windows.html) still > lists PCBuild and 2008. I should have checked first, but the readme.txt in the PCBuild directory seems to cover this in detail, so at worst, the online document is a minor issue. Although interestingly, the online doc points to PC/readme.txt which doesn't cover any of this. But I'm sure someone wanting to compile things is going to figure it out once they have the source in front of them. And it's still probably a good idea to have a PC/VS9.0 based builder. -- David From brett at python.org Fri Nov 18 18:16:41 2016 From: brett at python.org (Brett Cannon) Date: Fri, 18 Nov 2016 23:16:41 +0000 Subject: [Python-buildbots] Preparing for the GitHub migration Message-ID: As I begin to try and pin down final details for the GitHub migration in hopes of completing it by the end of December, I realized I needed to figure out how to migrate the buildbots. I assume there's two parts to this: (1) triggering the buildbots from GitHub, and (2) having the buildbots pull from GitHub. Does anyone know what needs to happen to make these two things occur? From greg at krypto.org Fri Nov 18 18:52:34 2016 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 18 Nov 2016 23:52:34 +0000 Subject: [Python-buildbots] Preparing for the GitHub migration In-Reply-To: References: Message-ID: I *believe* where and how the buildbots pull code from is controlled entirely by the buildbot master/server based on what I see my my own build slave's log. So switching is likely easy from that standpoint. One thing will be that people need to make sure their buildbots have git and any other tools necessary installed already. -gregory.p.smith On Fri, Nov 18, 2016 at 3:17 PM Brett Cannon wrote: > As I begin to try and pin down final details for the GitHub migration in > hopes of completing it by the end of December, I realized I needed to > figure out how to migrate the buildbots. I assume there's two parts to > this: (1) triggering the buildbots from GitHub, and (2) having the > buildbots pull from GitHub. > > Does anyone know what needs to happen to make these two things occur? > _______________________________________________ > Python-Buildbots mailing list > Python-Buildbots at python.org > https://mail.python.org/mailman/listinfo/python-buildbots > From rdmurray at bitdance.com Fri Nov 18 23:21:47 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 18 Nov 2016 23:21:47 -0500 Subject: [Python-buildbots] Preparing for the GitHub migration In-Reply-To: References: Message-ID: <20161119042147.7C9851B10003@webabinitio.net> On Fri, 18 Nov 2016 23:52:34 +0000, "Gregory P. Smith" wrote: > I *believe* where and how the buildbots pull code from is controlled > entirely by the buildbot master/server based on what I see my my own build > slave's log. So switching is likely easy from that standpoint. Correct, the slave polls the master for jobs, but the master tells the slave exactly what to do. So either Zach or I need to rewrite the master.cfg on the buildbot master to use git instead of hg. However, I don't actually remember how the master finds out about changes, so I'd have to look that up. Maybe Zach remembers. > One thing will be that people need to make sure their buildbots have git > and any other tools necessary installed already. Yes. We need to update that part of the devguide, as well. --David From zachary.ware+pydev at gmail.com Mon Nov 21 11:27:36 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Mon, 21 Nov 2016 10:27:36 -0600 Subject: [Python-buildbots] Preparing for the GitHub migration In-Reply-To: <20161119042147.7C9851B10003@webabinitio.net> References: <20161119042147.7C9851B10003@webabinitio.net> Message-ID: On Fri, Nov 18, 2016 at 10:21 PM, R. David Murray wrote: > On Fri, 18 Nov 2016 23:52:34 +0000, "Gregory P. Smith" wrote: >> I *believe* where and how the buildbots pull code from is controlled >> entirely by the buildbot master/server based on what I see my my own build >> slave's log. So switching is likely easy from that standpoint. > > Correct, the slave polls the master for jobs, but the master tells > the slave exactly what to do. So either Zach or I need to rewrite > the master.cfg on the buildbot master to use git instead of hg. Yep. I think buildbot has something specifically for tying into GitHub, but I'm not sure if our master is new enough to have it. We're a bit past due for an update there anyway, so I'm going to try to find time this weekend to both upgrade our master and try out GitHub integration. Changing the source for the builders will be simple once the master is looking at GitHub instead of hg.python.org. As a side note here, how interested are we in upgrading to buildbot 0.9.x? (See https://nine.buildbot.net/ for an example testing itself.) That upgrade will be a much larger project than the GitHub switch, but I've been wanting to rewrite the master anyway, and that would be a good opportunity to do the 0.9 upgrade. > However, I don't actually remember how the master finds out about > changes, so I'd have to look that up. Maybe Zach remembers. We have a hook on hg.python.org that pings the build master when changes are pushed. I'm not sure exactly how GitHub integration will work, but since buildbot itself is on GitHub I'm sure it's well-supported. -- Zach From dje.gcc at gmail.com Mon Nov 21 11:35:03 2016 From: dje.gcc at gmail.com (David Edelsohn) Date: Mon, 21 Nov 2016 11:35:03 -0500 Subject: [Python-buildbots] Preparing for the GitHub migration In-Reply-To: References: <20161119042147.7C9851B10003@webabinitio.net> Message-ID: On Mon, Nov 21, 2016 at 11:27 AM, Zachary Ware wrote: > On Fri, Nov 18, 2016 at 10:21 PM, R. David Murray wrote: >> On Fri, 18 Nov 2016 23:52:34 +0000, "Gregory P. Smith" wrote: >>> I *believe* where and how the buildbots pull code from is controlled >>> entirely by the buildbot master/server based on what I see my my own build >>> slave's log. So switching is likely easy from that standpoint. >> >> Correct, the slave polls the master for jobs, but the master tells >> the slave exactly what to do. So either Zach or I need to rewrite >> the master.cfg on the buildbot master to use git instead of hg. > > Yep. I think buildbot has something specifically for tying into > GitHub, but I'm not sure if our master is new enough to have it. > We're a bit past due for an update there anyway, so I'm going to try > to find time this weekend to both upgrade our master and try out > GitHub integration. Changing the source for the builders will be > simple once the master is looking at GitHub instead of hg.python.org. > > As a side note here, how interested are we in upgrading to buildbot > 0.9.x? (See https://nine.buildbot.net/ for an example testing > itself.) That upgrade will be a much larger project than the GitHub > switch, but I've been wanting to rewrite the master anyway, and that > would be a good opportunity to do the 0.9 upgrade. > >> However, I don't actually remember how the master finds out about >> changes, so I'd have to look that up. Maybe Zach remembers. > > We have a hook on hg.python.org that pings the build master when > changes are pushed. I'm not sure exactly how GitHub integration will > work, but since buildbot itself is on GitHub I'm sure it's > well-supported. Note that GDB tried to upgrade to the latest builtbot and it was a spectacular disaster. The new master is much more complicated and computationally-expensive in an effort to provide a more beautiful UI. Without a much more powerful system on which to run master, the information from the buildbot became useless. The buildbot also started to fall behind. Sergio started to discuss these problems with the buildbot developers. Please approach a general upgrade to the latest and greatest release of Buildbot master with a LOT of caution. Thanks, David From zachary.ware+pydev at gmail.com Mon Nov 21 11:51:36 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Mon, 21 Nov 2016 10:51:36 -0600 Subject: [Python-buildbots] Preparing for the GitHub migration In-Reply-To: References: <20161119042147.7C9851B10003@webabinitio.net> Message-ID: On Mon, Nov 21, 2016 at 10:35 AM, David Edelsohn wrote: > Note that GDB tried to upgrade to the latest builtbot and it was a > spectacular disaster. The new master is much more complicated and > computationally-expensive in an effort to provide a more beautiful UI. > Without a much more powerful system on which to run master, the > information from the buildbot became useless. The buildbot also > started to fall behind. That's very useful information, thank you. > Sergio started to discuss these problems with the buildbot developers. > > Please approach a general upgrade to the latest and greatest release > of Buildbot master with a LOT of caution. Right, my immediate plans are to upgrade to 0.8.14, which if I understand correctly should be relatively painless. -- Zach From zachary.ware+pydev at gmail.com Mon Nov 28 01:05:18 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Mon, 28 Nov 2016 00:05:18 -0600 Subject: [Python-buildbots] Preparing for the GitHub migration In-Reply-To: References: <20161119042147.7C9851B10003@webabinitio.net> Message-ID: On Mon, Nov 21, 2016 at 10:27 AM, Zachary Ware wrote: > We're a bit past due for an update [of the master], so I'm going to try > to find time this weekend to both upgrade our master and try out > GitHub integration. Changing the source for the builders will be > simple once the master is looking at GitHub instead of hg.python.org. Phase one of this plan is complete, the master is updated to buildbot 0.8.14+ (0.8.14 isn't actually tagged in github.com/buildbot/buildbot, so I just took the head of their 'eight' branch and called it 0.8.14py1). It's a fairly clean slate; I opted not to keep any of the existing history, though it's still available for now if I'm overruled. I forward-ported most of our patches, but at least one of them did not survive the transition from 'categories' to 'tags', so builder ordering is significantly different in just about every view now. I haven't had a chance to do any testing with GitHub integration, but it should be much simpler with an up-to-date master. I'll try to take a shot at that next weekend, along with some other tweaks to the master config that are now possible with 0.8.14. -- Zach From brett at python.org Mon Nov 28 13:07:24 2016 From: brett at python.org (Brett Cannon) Date: Mon, 28 Nov 2016 18:07:24 +0000 Subject: [Python-buildbots] Preparing for the GitHub migration In-Reply-To: References: <20161119042147.7C9851B10003@webabinitio.net> Message-ID: Thanks, Zach! On Sun, 27 Nov 2016 at 22:05 Zachary Ware wrote: > On Mon, Nov 21, 2016 at 10:27 AM, Zachary Ware > wrote: > > We're a bit past due for an update [of the master], so I'm going to try > > to find time this weekend to both upgrade our master and try out > > GitHub integration. Changing the source for the builders will be > > simple once the master is looking at GitHub instead of hg.python.org. > > Phase one of this plan is complete, the master is updated to buildbot > 0.8.14+ (0.8.14 isn't actually tagged in github.com/buildbot/buildbot, > so I just took the head of their 'eight' branch and called it > 0.8.14py1). It's a fairly clean slate; I opted not to keep any of the > existing history, though it's still available for now if I'm > overruled. I forward-ported most of our patches, but at least one of > them did not survive the transition from 'categories' to 'tags', so > builder ordering is significantly different in just about every view > now. > > I haven't had a chance to do any testing with GitHub integration, but > it should be much simpler with an up-to-date master. I'll try to take > a shot at that next weekend, along with some other tweaks to the > master config that are now possible with 0.8.14. > > -- > Zach > _______________________________________________ > Python-Buildbots mailing list > Python-Buildbots at python.org > https://mail.python.org/mailman/listinfo/python-buildbots >