[Distutils] Travis-CI is not open source. Was: Current Python packaging status (from my point of view)

[Thomas Güttler]

Am 01.11.2016 um 17:50 schrieb Matthew Brett:
> Hi,
> 
> On Tue, Nov 1, 2016 at 10:35 AM, Thomas Güttler
> <guettliml at thomas-guettler.de> wrote:
>>
>>
>> Am 01.11.2016 um 10:50 schrieb Nick Coghlan:
>>>
>>> On 1 November 2016 at 17:30, Thomas Güttler
>>> <guettliml at thomas-guettler.de> wrote:
>>>>
>>>> Am 17.09.2016 um 12:29 schrieb Nick Coghlan:
>>>>>
>>>>> Hi folks,
>>>>>
>>>>> Prompted by a few posts I read recently about the current state of the
>>>>> Python packaging ecosystem, I figured it made sense to put together an
>>>>> article summarising my own perspective on the current state of things:
>>>>>
>>>>> http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html
>>>>
>>>>
>>>>
>>>> Thank you for this summarizing article. Yes, a lot was done during the
>>>> last months.
>>>>
>>>> I liked the part "My core software ecosystem design philosophy" a lot,
>>>> since
>>>> it explains that both parties (software consumer and software publisher)
>>>> want
>>>> it to be simple and easy.
>>>>
>>>>
>>>> About conda: if pip and conda overlap in some point. Why not implement
>>>> this in a reusable library which gets used by conda and pip?
>>>
>>>
>>> For the parts where they genuinely overlap, conda is already able to
>>> just use pip, or else the same libraries that pip uses. For the
>>> platform management pieces (SAT solving for conda repositories,
>>> converting PyPI packages to conda ones, language independent
>>> environment management), what conda does is outside the scope of what
>>> pip supports anyway.
>>>
>>>> About funding: Looking for more income is one way to solve this. Why not
>>>> look into the other direction: How to reduce costs?
>>>
>>>
>>> Thanks to donated infrastructure, the direct costs to the PSF are
>>> incredibly low already. Donald went into some detail on that in
>>> https://caremad.io/posts/2016/05/powering-pypi/ and that's mostly
>>> still accurate (although his funded role with HPE ended recently)
>>>
>>>> Heading "Making the presence of a compiler on end user systems optional".
>>>> Here
>>>> I just can say: Thank you very much. I guess it was a lot of hard work to
>>>> make this all simple and easy for the software consumers and publishers.
>>>> Thank you.
>>>>
>>>> I wrote some lines, but I deleted my thoughts about the topic "Automating
>>>> wheel creation", since
>>>> I am a afraid it could raise bad mood in this list again. That's not my
>>>> goal.
>>>
>>>
>>> I currently see 3 main ways that could eventually happen:
>>>
>>> - the PSF sorts out its general sustainability concerns to the point
>>> where it believes it can credibly maintain such a service on the
>>> community's behalf
>>> - the conda-forge folks branch out into offering wheel building as
>>> well (so it becomes a matter of "publish your Python projects for the
>>> user level conda platform, get platform independent Python wheels as
>>> well")
>>> - someone builds such a service independently of the current PyPI
>>> infrastructure team, and convinces package publishers to start using
>>> it
>>>
>>> There's also a 4th variant, which is getting to a point where someone
>>> figures out a pushbutton solution for a build pipeline in a public
>>> PaaS that offers a decent free tier. This is potentially one of the
>>> more promising options, since it means the sustainability risks
>>> related to future growth in demand accrue to the PaaS providers,
>>> rather than to the PSF. However, it's somewhat gated on the Warehouse
>>> migration, since you really want API token support for that kind of
>>> automation, which is something the current PyPI code base doesn't
>>> have, and isn't going to gain.
>>
>>
>> I like this 4th variant. I guess most companies which use Python
>> in a professional way already run a own pypi server.
>>
>> I am unsure if a public PaaS for this would exist. Maybe a script
>> which runs a container on linux is enough. At least enough to
>> build linux-only wheels. I guess most people have root access
>> to a linux server somewhere.
> 
> I wrote some scripts to do this:
> 
> https://github.com/matthew-brett/multibuild
> 
> See the README there for standard usage.   For simple projects the
> project configuration is only a few lines.  Quite a few projects use
> multibuild and travis-ci to build 32- and 64-bit manylinux and OSX
> wheels, here are some (more complex) examples:
> 
> https://travis-ci.org/MacPython/numpy-wheels
> https://travis-ci.org/MacPython/scipy-wheels
> https://travis-ci.org/MacPython/matplotlib-wheels
> https://travis-ci.org/scikit-image/scikit-image-wheels
> https://travis-ci.org/MacPython/pandas-wheels
> https://travis-ci.org/python-pillow/pillow-wheels
> https://travis-ci.org/MacPython/cython-wheels
> 
> The travis-ci jobs mostly upload to a CDN on a Rackspace account
> donated to scikit-learn, from which the release managers can upload to
> pypi with a few lines at the terminal.
> 
> So, for many of us scientific Python projects, and for Linux and OSX,
> this problem has a reasonable solution in principle.
> 
> One hurdle at the moment, is that new projects have to either get an
> encrypted key to the scikit-learn account from one of us with the
> credentials, or roll their own mechanism of uploading the built
> wheels.  It would be very helpful to have some standard pypa account
> mechanism for this.   That would allow us to standardize the tools and
> permission mechanism,


I see an other hurdle. travis-ci is very widespread, but AFAIK it is not open source:

   https://travis-ci.com/plans



-- 
http://www.thomas-guettler.de/