[Python-ideas] Python-specific Travis CI

[David Wilson]

Hi there,

In my travails to get py-lmdb (github.com/dw/py-lmdb) into a state I can 
depend on, I've been exploring continuous integration options since 
sometime mid last year, in the hopes of finding the simplest option that 
allows the package to be tested on all sensible versions of Python that 
I might bump into in a commercial environment.

It seems this simple task currently yields one solution - Travis CI, 
except that by default Travis supports neither Windows, nor any version 
of Python older than about 3 months, without requiring serious manual 
configuration and maintenance.

The library just about has a working Travis CI at this point – although 
it is incredibly ugly. Testing against all recent versions of Python (by 
"recent" I mean "anything I've seen and might reasonably expect to bump 
into while consulting, i.e. since 2.5") requires around 14 Debian 
packages to be installed, followed by installation of compatible 
versions of easy_install and Pip for each interesting Python version.

The result just about works, but this week I'm again finding that it 
doesn't quite go far enough, as I begin working on a Windows binary 
distribution for my package.

Windows is almost the same problem all over again - testing 
supported/sensible extension module builds requires 14 binary 
installations of Python (one for each of 32 bit and 64 bit), 3 distinct 
versions of Microsoft Visual Studio (that I'm aware of so far), and a 
compatible version of Windows to run it all on.

So the work done to support CI on Travis more or less isn't enough. It 
seems the best option now is a custom Jenkins setup, and acquiring 
licenses for all the Microsoft software necessary to build the package.


You may wonder why one might care about so many old Python versions in 
weird and exotic environments - to which I can't really give a better 
answer than that I've lost so much time working with packages that 
suffer from platform/version compatibility issues, that I'd like to 
avoid it in my own code.

While it isn't so true in web development, there are pretty huge 
installations of Python around the world, many of which aren't quite 
near migrating to 2.6, never mind 3.4. This can happen for different 
reasons, from maintaining private forks of Python (perhaps in a 
post-audited state, where another audit simply isn't in the budget), 
through to having thousands of customers who have written automation 
scripts for their applications using an older dialect.

In any case it's enough to note these installations do and will continue 
to exist, than to question why they exist..


So in the coming months I'll probably end up configuring a Jenkins, 
porting the Travis CI scripts over to it to a private Ubuntu VM 
configured as a Jenkins runner, and setting up a new Windows VM as a 
second runner, but it seems entirely wasteful that so much effort should 
be made for just one project.

And that leads to thinking about a Python-specific CI service, that 
could support builds similar to Travis CI, except for all configurations 
of Python still in active use. One aspect of such a service that is 
particularly interesting, is the ability to centralize management of the 
required licenses – the PSF has a much greater reputation, and is much 
more likely to be granted a blanket license covering a CI service for 
all open source Python code, than each individual project is by applying 
individually.

Licensing aside, having a 0-cost option for any Python project to be 
tested would improve the quality of our ecosystem overall, especially in 
the middle of the largest transition to date - the 3.x series.

Following from the ecosystem improvement angle, there might be no reason 
a Python-specific CI service could not automatically download, build and 
py.test every new package entering PyPI, producing as side effect 
scoreboards similar to the Python 3 Wall of Superpowers 
(http://python3wos.appspot.com/). This would make a sound base for 
building future improvements for PyPI - for example a lint service, 
verifying "setup.py" is not writing to /etc/passwd, or uploaded eggs 
aren't installing a new SVCHOST.EXE to %SystemDir%.

It could even make sense to have the hypothetical service produce 
signed, "golden" artefacts for upload to PyPI - binaries built from 
source in a trusted environment, free of any Bitcoin ransom-ware 
surreptitiously inserting itself into the bdist_egg ZIP file.

There are many problems with providing a CI service, for example, the 
handling of third party commercial libraries required during the build 
process. An example of this might be producing a build of cx_Oracle, 
where Oracle's proprietary SDK is a prerequisite.

Still, it would not require many resources, or even very much code, to 
provide a service that could minimally cover at least basic needs for 
purely open source software, and so the idea appeals to me.

Any thoughts?


David