[Distutils] Binary Wheels and "universal" builds on OS-X

[Chris Barker - NOAA Federal]

On Fri, May 31, 2013 at 3:38 PM, Daniel Holth <dholth at gmail.com> wrote:
> Wheel already supports compound tags. Just like py2.py3-none-any a tag

Is this from PEP 425? It's still a little unclear to me exactly how
those tags are to be used, but, yes, this seems to be the right
direction.

> py3-none-x86.ppc (with real platform names) would work. Does that make sense
> for Mac?

hmm, for one, I"m confused about the "Python Tag" vs. the "ABI Tag"
vs. the "Platform Tag" -- I guess these all get mushed together in the
filename?

but a few issues anyway: it looks like the platform tag has the OS and
a architecture in one, so would a universal bild on the mac be:

macosx_10_3_i386.ppc
or
macosx_10_3_i386.macosx_10_3_ppc

I'm thinking the latter, though it gets pretty wordy:

A potential full name:

my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl

(Im a little confused about python version tag vs. ABI tag, in the
case of a compiled extension...)

But a potential confusion -- in this case, having two platform tags
indicates that _both_ are present, rather than one thing that works
with either. Maybe that would always be the case with platform.

So if that's how the wheel is specified, we still need the other side
of the process:

Installing:

A "universal" macpython binary needs to know that it is universal
(distutils.util.get_platform() returns only the currently running
version). Once that is know, the installer needs to make some choice:

If a matching universal binary is found -- use it
If a non-universal binary is found that matches the currently running
python -- then what? install it with a warning? raise an exception?

-Chris










>
> On May 31, 2013 6:30 PM, "Chris Barker - NOAA Federal"
> <chris.barker at noaa.gov> wrote:
>>
>> HI Folks,
>>
>> A few of us over on the pythonmac list have been hashing out how best
>> to support binary packages on OS-X. The binary wheel option seems very
>> promising.
>>
>> However, there is one Mac-specific issue that does not seem to be
>> addressed:
>>
>> On OS-X, binaries can be "universal" -- what this means is that they
>> have more than one binary architecture embedded in a single file; the
>> system selects the right one at run time. Both executables and dynamic
>> libraries can be universal. In theory, an universal binary can
>> currently contain up to 4 architectures:
>>
>> 32+64 bit PPC and 32+64bit x86
>>
>> In practice, 64 bit PPC was never broadly used, and 32bit PPC is
>> fading fast. Nevertheless, both 32 and 64 Intel systems are pretty
>> common, and who knows what there may be in the future (you never know
>> with Apple...)
>>
>> The binary builds of Python served up on the python.org web site have
>> supported universal builds for years -- currently there are two
>> options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it
>> turns out it's hard to support both up to date x86_64 and PPC with the
>> same compiler. In these builds, the configuration is set up so that
>> distutils will build universal extensions that match the architectures
>> included in the builds. This makes it pretty easy to build binary
>> packages, and/or full app distributions (py2app) that are universal as
>> well.
>>
>> setuptools' easy_install supports binary eggs, but always choked on
>> universal builds -- it didn't know how to identify what matched the
>> system. It would be really nice if future tools could identify and
>> install the correct binary wheels for universal builds.
>>
>> I've taken a look at PEP 427, and see this:
>>
>> platform tag
>> E.g. 'linux_x86_64', 'any'.
>>
>> That looks to me like there is a built-in assumption that a wheel is
>> either specific to a single architecture, or any -- so no way to
>> specify that it may have multiple architectures. So I propose that
>> platform tag allow a list of some sort:
>>
>> ['macosx-10_6_i386', 'macosx_10_6-x86_64']
>>
>> or, I suppose to keep it simple, a single string:
>>
>> 'macosx_10_6_i386+macosx_10_6_x86_64'
>> or
>> 'macosx_10_6_i386+x86_64'
>>
>> If so, then a binary wheel could be built (and discovered) that
>> supported multiple architectures.
>>
>> Then how would an installer ( pip? ) identify the right one? This is a
>> bit tricky. PEP 425 states:
>>
>> "The platform tag is simply distutils.util.get_platform() with all
>> hyphens - and periods . replaced with underscore"
>>
>> On my system right now, I have 32bitPPC + 32bit i386 universal python,
>> running on an intel system:
>>
>> In [2]: distutils.util.get_platform()
>> Out[2]: 'macosx-10.3-i386'
>>
>> So we may want to patch get_platform() to support universal builds -
>> or add anther function or something -- you may sometimes what to know
>> what you are running right now, and other times want to know how the
>> current python is built. (note that while distutils can build
>> universal binaries, it's not very smart about it -- all is does is
>> grab the compiler and linker flags from the Makefile, which include
>> multiple "-arch" flags.
>>
>> Even if there is an easy way to query the system, that leaves the
>> question of what to do. If a user wants to intall a package into a
>> universal python, will the installer only accept a binary wheel with
>> all the supported architectures? or one with only the currently
>> running one?
>>
>> Anyway, it would be nice to be able to get this stuff to "just work"
>> with universal binaries on the Mac.
>>
>> Thoughts, ideas?
>>
>> -Chris
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov