[Distutils] Working toward Linux wheel support
Marcus Smith
qwcode at gmail.com
Fri Jul 17 18:50:17 CEST 2015
I think Linux wheel support is almost useless unless the pypa stack
> provides _something_ to handle non-python dependencies.
>
I wouldn't say useless, but I tend to agree with this sentiment.
I'm thinking the only way to really "compete" with the ease of Conda (for
non-python dependencies) is to shift away from wheels, and instead focus on
making it easier to create native distro packages (i.e. rpm, deb etc...that
can easily depend on non-python dependencies) for python applications, and
moreover that these packages should be "parallel installable" with the
system packages, i.e. they should depend on virtual environments, not the
system python.
I've been working on this some personally, admittedly pretty slowly, since
it's a pretty tall order to put all the pieces together
Marcus
>
> 1) Pure Python packages work fine as source.
>
> 2) Python packages with C extensions build really easily out of the box
> -- so source distribution is fine (OK, I suppose some folks want to
> run a system without a compiler -- is this the intended use-case?)
>
> So what are the hard cases? the one we really want binary wheels for?
>
> - Windows, where a system compiler is a rarity: Done
>
> - OS-X, where a system compiler is a semi-rarity, and way too many
> "standard" system libs aren't there (or are old crappy versions...) -
> Almost Done.
>
> - Packages with semi-standard dependencies: can we expect ANY Linux
> distro to have libfreetype, libpng, libz, libjpeg, etc? probably, but maybe
> not installed (would a headless server have libfreetype?). And would those
> version be all compatible (probably if you specified a distro version)
>
> - Packages with non-standard non-python dependencies: libhdf5, lapack,
> BLAS, fortran(!) -- this is where the nightmare really is. I suspect most
> folks on this list will say that this is "Scipy Problem", and indeed,
> that's where the biggest issues are, and where systems like conda have
> grown up to address this.
>
> But at this point, I think it's really sad that the community has become
> fractured -- if folks start out with "I want to do scientific computing",
> then they get pointed to Enthought Canopy or Anaconda, and all is well
> (until they look for standard web development packages -- though that's
> getting better). But if someone starts out as a web developer, and is all
> happy with the PyPA stack (virtualenv, pip, etc...), then someone suggests
> they put some Bokeh plotting in their web site, or need to do
> some analytics on HDF5 files, or any number of things well supported by
> Python, but NOT by pip/wheel -- they are kind of stuck.
>
> My point is that it may actually be a bad thing to solve the easy problem
> while keeping out fingers in our ears about the hard ones....
>
> (la la la la, I don't need to use those packages. la la la la)
>
> My thought: what pip+wheel needs to support much of this is the ability to
> specify a wheel dependency, rather than a package dependency -- i.e. "this
> particular wheel requires a libfreetype wheel". Then we could have binary
> wheels for non-python dependencies like libs (which would install the lib
> into pre-defined locations that could be relative path linked to)
>
> Sorry for the rant....
>
> -Chris
>
> PS: Personally, after banging my head against this for years,
> I've committed to conda for the moment -- working to get conda to better
> support the wide range of python packages. I haven't tried it on Linux, but
> it does exist and works well for some folks.
>
>
>
> On Fri, Jul 17, 2015 at 1:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 17 July 2015 at 03:41, Nate Coraor <nate at bx.psu.edu> wrote:
>> > Hi all,
>> >
>> > I've recently been working on adding SOABI support for Python 2.x and
>> other
>> > pieces needed to get wheels w/ C extensions for Linux working. Here's
>> the
>> > work for wheels:
>> >
>> > https://bitbucket.org/pypa/wheel/pull-request/54/
>> >
>> > Based on that, I've added support for those wheels to pip here:
>> >
>> > https://github.com/natefoo/pip/tree/linux-wheels
>>
>> >
>> > As mentioned in the wheels PR, there are some questions and decisions
>> made
>> > that I need guidance on:
>> >
>> > - On Linux, the distro name/version (as determined by
>> > platform.linux_distribution()) will be appended to the platform string,
>> e.g.
>> > linux_x86_64_ubuntu_14_04. This is going to be necessary to make a
>> > reasonable attempt at wheel compatibility in PyPI. But this may violate
>> PEP
>> > 425.
>>
>> I think it's going beyond it in a useful way, though. At the moment,
>> the "linux_x86_64" platform tag *under*specifies the platform - a
>> binary extension built on Ubuntu 14.04 with default settings may not
>> work on CentOS 7, for example.
>>
>> Adding in the precise distro name and version number changes that to
>> *over*specification, but I now think we can address that through
>> configuration settings on the installer side that allow the
>> specification of "compatible platforms". That way a derived
>> distribution could add the corresponding upstream distribution's
>> platform tag and their users would be able to install the relevant
>> wheel files by default.
>>
>> Rather than putting the Linux specific platform tag derivation logic
>> directly in the tools, though, what if we claimed a file under the
>> "/etc/python" subtree and used it to tell the tools what platform tags
>> to use? For example, we could put the settings relating to package
>> tags into "/etc/python/binary-compatibility.cfg" and allow that to be
>> overridden on a per-virtualenv basis with a binary-compatibility.cfg
>> file within the virtualenv.
>>
>> For example, we could have a section where for a given platform, we
>> overrode both the build and install tags appropriately. For RHEL 7.1,
>> that may look like:
>>
>> [linux_x86_64]
>> build=rhel_7_1
>> install=rhel_7_0,rhel_7_1,centos_7_1406,centos_7_1503
>>
>> Using JSON rather than an ini-style format would also work:
>>
>> {
>> "linux_x86_64": {
>> "build": "rhel_7_1",
>> "install": ["rhel_7_0", "rhel_7_1", "centos_7_1406",
>> "centos_7_1503"]
>> }
>> }
>>
>> The reason I like this approach is that it leaves the definition of
>> ABI compatibility in the hands of the distros, but also makes it safe
>> to publish Linux wheel files on PyPI (just not with the generic
>> linux_x86_64 platform tag).
>>
>> > - By default, wheels will be built using the most specific platform
>> > information. In practice, I build our wheels[1] using Debian Squeeze in
>> > Docker and therefore they should work on most currently "supported"
>> Linuxes,
>> > but allowing such wheels to PyPI could still be dangerous because
>> forward
>> > compatibility is not always guaranteed (e.g. if a SO version/name
>> changes,
>> > or a C lib API method changes in a non-backward compatible way but the
>> SO
>> > version/name does not change). That said, I'd be happy to make a much
>> more
>> > generalized version of our docker-build[2] system that'd allow package
>> > authors to easily/rapidly build distro/version-specific wheels for many
>> of
>> > the popular Linux distros. We can assume that a wheel built on a vanilla
>> > install of e.g. Ubuntu 14.04 will work on any other installation of
>> 14.04
>> > (this is what the distro vendors promise, anyway).
>>
>> Right, if we break ABI within a release, that's our fault (putting on
>> my distro developer hat), and folks will rightly yell at us for it. I
>> was previously wary of this approach due to the "what about derived
>> distributions?" problem, but realised recently that a config file that
>> explicitly lists known binary compatible platforms should suffice for
>> that. There's only a handful of systems folks are likely want to
>> prebuild wheels for (Debian, Ubuntu, Fedora, CentOS/RHEL, openSuse),
>> and a configuration file based system allows ABI compatible derived
>> distros to be handled as if they were their parent.
>>
>> > - I attempt to set the SOABI if the SOABI config var is unset, this is
>> for
>> > Python 2, but will also be done even on Python 3. Maybe that is the
>> wrong
>> > decision (or maybe SOABI is guaranteed to be set on Python 3).
>>
>> Python 3 should always set it, but if it's not present for some
>> reason, deriving it makes sense.
>>
>> > - Do any other implementations define SOABI? PyPy does not, I did not
>> test
>> > others. What should we do with these?
>>
>> The implementation identifier is also included in the compatibility
>> tags, so setting that in addition to the platform ABI tag when a wheel
>> contains binary extensions should suffice.
>>
>> > Because the project I work for[3] relies heavily on large number of
>> > packages, some of which have complicated build-time dependencies, we
>> have
>> > always provided them as eggs and monkeypatched platform support back in
>> to
>> > pkg_resources. Now that the PyPA has settled on wheels as the preferred
>> > binary packaging format, I am pretty heavily motivated to do the work to
>> > work out all the issues with this implementation.
>>
>> Thank you!
>>
>> Regards,
>> Nick.
>>
>> --
>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
>> _______________________________________________
>> Distutils-SIG maillist - Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150717/76d38e4d/attachment-0001.html>
More information about the Distutils-SIG
mailing list