[Cryptography-dev] Dedicated Travis Builders

[Paul Kehrer]

(Sorry for the exposition that's about to happen, just wanted to write it all down first)

We use two different CI systems right now:

- Travis, which provides OS X 10.9 builds as well as linux builds (against Ubuntu 12.04 with two different versions of OpenSSL).
- Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie, Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL, Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server 2012 (64-bit Pythons).

Travis provides us some (large) advantages over Jenkins, including not having to manage our own fleet of builders, far better (albeit less flexible) configuration through .travis.yml, and a difficult to quantify "pleasantness" to the entire experience.

So, what problems do we run into with Travis?

Coverage
------------
At this time we use coveralls (which is now commenting up to 9 times on every PR for reasons that pass human understanding) to gain a view of our combined coverage. Unfortunately this system ties us to reporting coverage exclusively from Travis. This means code paths (like windows only code) cannot have coverage tracked.

Speed/Reliability
----------------------
We have limited (albeit very high at the moment) concurrency available from Travis. On a typical workday it can be 1-3 hours before CI completes, which significantly impairs our ability to quickly review/merge code.  We have had occasions in the past where we've had Travis jobs waiting in the queue for over 8 hours due to reliability problems within the Travis infrastructure.

Flexibility
------------
Travis provides only OS X 10.9 and Ubuntu 12.04.



The openstack builder possibility solves much of the speed issue, and custom docker images would negate the need for our jenkins linux builders (of which there are currently 7). I am, however, concerned about coverage. Without being able to run more than OS X 10.9 and (potentially) various userlands of Linux via docker images we're still significantly handicapped. It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.

We have recently discussed moving entirely off Travis (https://github.com/pyca/cryptography/issues/1729), but if the above comes to fruition and there's a good way to combine coverage from multiple sources (and someone wants to own getting it deployed that isn't me) then I'd be happy to stay with Travis and only use jenkins for the edge cases.

-Paul

On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:

I was talking to the Travis CI folks and they mentioned they have the  
possibility of using openstack builders now (or in the near future). We talked  
a little bit about the possibility of getting PyCA it's own dedicated set of  
builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.  

Currently the entirety of the PyPA organization on Github gets 10 concurrent  
builders (typically this number is 5) across all repositories. Assuming Travis  
CI is able to do it (they'd need to check with their ops team, and there would  
be some cost associated with things on their side as well) we could essentially  
control how much concurrency we want by just throwing more Rackspace VMs at  
our builds.  

Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace  
Compute1-4 which is similar we'd be looking at the ability to run ~27 of those  
VMs full time on the "standard" 2k/month free cloud offering that Rackspace  
offers OSS projects. However we'd actually be able to get more than that  
probably because Travis starts up the VMs on demand so anytime there aren't  
tests to run we won't have any VMs running.  

Independent of this, they are also working on the ability to run custom docker  
images (which means custom Linux OS images) that we can preinstall different  
software into.  

Is this something we'd want to explore? Assuming that the Travis CI ops team  
and such are OK with it, it could essentially let us scale up our concurrency  
on Travis to whatever amount of dollars of Rackspace cloud we want to throw at  
it.  

---  
Donald Stufft  
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA  

_______________________________________________  
Cryptography-dev mailing list  
Cryptography-dev at python.org  
https://mail.python.org/mailman/listinfo/cryptography-dev  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/9f28558d/attachment-0001.html>