[Distutils] Plans for binary wheels, and PyPi and OS-X

[Chris Barker]

On Fri, Nov 1, 2013 at 6:59 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> The key point here is the granularity of the PEP 425 tags used by wheel.
>
> The risk is that a wheel created on another system might declare (via
> its filename) that it is compatible with your system, and then not be,
> causing segfaults or similar when used.


indeed, which is why it _might_ be a good idea to include an extra python
build flag or something: "python.org", "homebrew", "macports". However,
it's probably the case that those aren't really the issues that are going
to cause problems -- at least ones that aren't already handled by the OS-X
version flags --- i.e if a package is built for 10.6+, then it should have
the same system libs as a python built for 10.6+.

Practically speaking, the issues I've run into are:

* Packages built for a newer OS-X won't work on an older one -- but that
should be handled by the OS-X version handling that exists.

* "universal" binaries -- packages built for 32 bit aren't going to work
with a 64 bit python, and a universal python can be both 32 and 64 bit (and
PPC, but I think those days are gone...) -- but this _should_ be handled by
the platform flag: IIRC, "intel" means 32+64 bit Intel. Though I'm not sure
what homebrew or macports python report. But distutils generally does the
right thing with self-contained C code.

* External dependencies: This is the BIG ONE: it's the hardest to get
right, and the hardest to check for. Third party libs must:
  - Be built to match the python, including SDK and architecture (including
universal)
  - Be included somehow -- ideally statically linked, but I'm thinking that
they could be included as part of another dependent package (I think that's
how Anocanda does it). The trick with dynamic linking on OS-X is that that
standard way to install and link a lib has the path to the lib hard-coded
in -- so you can't move it without re-writing the headers. This can be done
on install, but I don't hink we want pip to have to deal with that! You
_can_ install and link libs with relative paths, which I think is what
Anaconda is doing, but I haven't figured out how yet, and it's certainly
not a no-brainer.

So I don't think there is any way to get around the fact that you need to
be careful to build a binary wheel that will work on the systems you
are targeting -- but this is no different than the situation we've had for
years with building binary installers for the Mac. But those dont work with
pip, or virtualenv, or...


> On Windows, thanks to the
> history of bdist_wininst, and the small number of people who build
> their own *anything* on Windows, there is really only one ABI/C
> Library/whatever to worry about and that is the python.org one.
> (Actually, there are two - 32 and 64 bit).
>

Well, technically, the situation is very similar -- it's hard to build a
Windows binary (at least if it has external dependencies), so that it will
just work.

Socially, the situation is different -- there are a (relatively) small
number of people building their own stuff. On the Mac, however, you have
homebrew and macports, and ??, so lots of people building their own stuff.
But those aren't the people we need to support with binaries!

Is anyone expecting a binary built for Windows to work with a cygwin python?
Is anyone expecting that they can build a binary on Windows with cygwin and
give it out? That's what we're talking about here with the Mac.

thanks to the history of bdist_wininst ..


The mac has a history of bdist_mpkg as well, not as widely used, and a bit
neglected lately, but it's there. And there is a history of folks providing
binary installers for the python.org mac. But it would be really nice if we
could go to wheels, and use pypi to distribute them.

It really is the same as Windows -- anyone putting a binary on PyPi has an
obligation to built is so it will work with the python.org python -- and
it's not inherently any harder to do that than on Windows -- the only
difference is that it may be easier to do it badly -- by simply running
bdist_wheel without thinking about it (i.e with homebrew, or macports and
whatever shared libs happen to be linked to).

But again, that's a social problem -- we need to have an understanding
about what is required to put a binary wheel up on pypi.

Also, where we _could_ have a way to identify python.org, vs homebrew, vs.
macports as different "platfroms", that's not going to help the hard
problem, which is making sure third party libs are built and included
properly.

>
> If all builds on OSX are compatible at the ABI/CLib/architecture level
> then there should be no problem. Equally, if incompatible builds
> result in wheels with different compatibility tags, also no problem.
> It's only if 2 incompatible environments use the same tags that there
> could be an issue.
>

yeah -- but the third party libs are the bigger issue anyway...


> I don't believe that linking with external libraries should be a
> problem here - if wheel X depends on library Y, then either it should
> include it or it should simply document "Needs library Y installed".
> It's not a *compatibility* issue as such.


no -- but it's the hard problem -- the POINT of providing binaries is for
people that don't know how to built it themselves to have something they
can use. There is no point at all in providing a binary python package that
depends on a particular freetype, or libpng, or whatever -- if you can
build those, you can build the package. (or have homebrew, etc. do it for
you).

But maybe the Unix concept
> of executables linking to libraries in hard-coded paths might be an
> issue here, I don't know enough about how that works to say.
>

well, that can be gotten around, but it's not really different -- if you
have a binary linked to a non-system shared lib, that shared lib needs to
be properly installed. Properly may be differently defined than on other
systems, but it still needs to be done.

NOTE: somewhere in this thread a saw anote about binary eggs not working
well with OS-X. AFAIK, the issue there is that setuptools did not
understand "universal" packages -- i.e. if you happened to be running ppc,
then it would look for a ppc egg, and not know a "universal" egg was what
it needed. IN fact, it would often install it correctly, but then not know
it had, and try to build it from source anyway.

But I think pip and wheel have got this right now -- but we won't know
'till we try!

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/02aaa145/attachment-0001.html>