[Cryptography-dev] Dedicated Travis Builders

[Alex Gaynor]

Sorry, I'm not following, it seems like you're assuming that "thing that
replaces coveralls" wouldn't be able to work with both Travis and Jenkins?

Alex

On Sun, Mar 29, 2015 at 12:59 PM, Paul Kehrer <paul.l.kehrer at gmail.com>
wrote:

> Travis is (potentially) offering to undertake significant work on our
> behalf so I believe we need to have an answer to our long term coverage
> questions before we commit to using this solution. Being faster and having
> custom docker images doesn't grant us a workable coverage solution so we're
> going to be discussing moving entirely to jenkins again at some point in
> the future if we can't resolve that issue.
>
> -Paul
>
> On March 29, 2015 at 11:55:55 AM, Alex Gaynor (alex.gaynor at gmail.com)
> wrote:
>
> Paul, it seems like coveralls is kind of orthagonal to this porposal,
> besides the fact that in the short term moving more stuff under Travis
> would mean coveralls included more stuff?
>
> Alex
>
> On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer <paul.l.kehrer at gmail.com>
> wrote:
>
>>  (Sorry for the exposition that's about to happen, just wanted to write
>> it all down first)
>>
>> We use two different CI systems right now:
>>
>> - Travis, which provides OS X 10.9 builds as well as linux builds
>> (against Ubuntu 12.04 with two different versions of OpenSSL).
>> - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie,
>> Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL,
>> Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X
>> 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server
>> 2012 (64-bit Pythons).
>>
>> Travis provides us some (large) advantages over Jenkins, including not
>> having to manage our own fleet of builders, far better (albeit less
>> flexible) configuration through .travis.yml, and a difficult to quantify
>> "pleasantness" to the entire experience.
>>
>> So, what problems do we run into with Travis?
>>
>> Coverage
>> ------------
>> At this time we use coveralls (which is now commenting up to 9 times on
>> every PR for reasons that pass human understanding) to gain a view of our
>> combined coverage. Unfortunately this system ties us to reporting coverage
>> exclusively from Travis. This means code paths (like windows only code)
>> cannot have coverage tracked.
>>
>> Speed/Reliability
>> ----------------------
>> We have limited (albeit very high at the moment) concurrency available
>> from Travis. On a typical workday it can be 1-3 hours before CI completes,
>> which significantly impairs our ability to quickly review/merge code.  We
>> have had occasions in the past where we've had Travis jobs waiting in the
>> queue for over 8 hours due to reliability problems within the Travis
>> infrastructure.
>>
>> Flexibility
>> ------------
>> Travis provides only OS X 10.9 and Ubuntu 12.04.
>>
>>
>>
>> The openstack builder possibility solves much of the speed issue, and
>> custom docker images would negate the need for our jenkins linux builders
>> (of which there are currently 7). I am, however, concerned about coverage.
>> Without being able to run more than OS X 10.9 and (potentially) various
>> userlands of Linux via docker images we're still significantly handicapped.
>> It is unreasonable to expect to be able to run arbitrary OSes so we'll
>> always run a small jenkins instance for things like alternate OS X
>> versions, FreeBSD, etc, but the lack of Windows support is painful.
>>
>> We have recently discussed moving entirely off Travis (
>> https://github.com/pyca/cryptography/issues/1729), but if the above
>> comes to fruition and there's a good way to combine coverage from multiple
>> sources (and someone wants to own getting it deployed that isn't me) then
>> I'd be happy to stay with Travis and only use jenkins for the edge cases.
>>
>> -Paul
>>
>> On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:
>>
>>   I was talking to the Travis CI folks and they mentioned they have the
>> possibility of using openstack builders now (or in the near future). We
>> talked
>> a little bit about the possibility of getting PyCA it's own dedicated set
>> of
>> builders in Travis CI hosted inside of Rackspace on one of our cloud
>> accounts.
>>
>> Currently the entirety of the PyPA organization on Github gets 10
>> concurrent
>> builders (typically this number is 5) across all repositories. Assuming
>> Travis
>> CI is able to do it (they'd need to check with their ops team, and there
>> would
>> be some cost associated with things on their side as well) we could
>> essentially
>> control how much concurrency we want by just throwing more Rackspace VMs
>> at
>> our builds.
>>
>> Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the
>> Rackspace
>> Compute1-4 which is similar we'd be looking at the ability to run ~27 of
>> those
>> VMs full time on the "standard" 2k/month free cloud offering that
>> Rackspace
>> offers OSS projects. However we'd actually be able to get more than that
>> probably because Travis starts up the VMs on demand so anytime there
>> aren't
>> tests to run we won't have any VMs running.
>>
>> Independent of this, they are also working on the ability to run custom
>> docker
>> images (which means custom Linux OS images) that we can preinstall
>> different
>> software into.
>>
>> Is this something we'd want to explore? Assuming that the Travis CI ops
>> team
>> and such are OK with it, it could essentially let us scale up our
>> concurrency
>> on Travis to whatever amount of dollars of Rackspace cloud we want to
>> throw at
>> it.
>>
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>>  ------------------------------
>> _______________________________________________
>> Cryptography-dev mailing list
>> Cryptography-dev at python.org
>> https://mail.python.org/mailman/listinfo/cryptography-dev
>>
>>
>> _______________________________________________
>> Cryptography-dev mailing list
>> Cryptography-dev at python.org
>> https://mail.python.org/mailman/listinfo/cryptography-dev
>>
>>
>
>
> --
>  "I disapprove of what you say, but I will defend to the death your right
> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084
>   _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev
>
>
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev
>
>


-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/f43dd743/attachment.html>