[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau <cournape at gmail.com> wrote:
>
>
> On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
>>
>> NOTE: I don't work for any of the companies involved -- just a somewhat
>> frustrated user... And someone that has been trying for years to make things
>> easier for OS-X users.
>>
>>>> I’m not sure what (3) means exactly. What is a “normal” Python, do you
>>>> modify Python in a way that breaks the ABI but which isn’t reflected in the
>>>> standard ABI tag?
>>>
>>>
>>> It could be multiple things. The most obvious one is that generally.
>>> cross-platforms python distributions will try to be "relocatable" (i.e. the
>>> whole installation can be moved and still work). This means they require
>>> python itself to be built a special way. Strictly speaking, it is not an ABI
>>> issue, but the result is the same though: you can't use libraries from
>>> anaconda or canopy on top of a normal python
>>
>>
>> But why not? -- at least for Anaconda, it's because those libraries likely
>> have non-python dependencies, which are expected to be installed in a
>> particular way. And really, this is not particular to Anaconda/Canopy at
>> all. Python itself has no answer for this issue, and eggs and wheels don't
>> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to
>> build a binary wheel with non-python dependencies (let's say something like
>> libjpeg, for instance), you need to either:
>>  - assume that libjpeg is installed in a "standard" place -- really no
>> solution at all (at least outside of linux)
>>  - statically link it
>>  - ship the dynamic lib with the package
>>
>> For the most part, the accepted solution for OS-X has been to statically
>> link, but:
>>
>>  - it's a pain to do. The gnu toolchain really likes to use dynamic
>> linking, and building a static lib that will run on a
>> maybe-older-than-the-build-system machine is pretty tricky.
>>
>>  - now we end up with multiple copies of the same lib in the python
>> install. There are a handful of libs that are used a LOT. Maybe there is no
>> real downside -- disk space and memory are cheap these days, but it sure
>> feels ugly. And I have yet to feel comfortable with having multiple versions
>> of the same lib linked into one python instance -- I can't say I've seen a
>> problem, but it makes me nervous.
>>
>> On Windows, the choices are the same, except that: It is so much harder to
>> build many of the "standard" open source libs that package authors are more
>> likely to do it for folks, and you do get the occasional "dll hell" issues.
>>
>> I had a plan to make some binary wheels for OS-X that were not really
>> python packages, but actually just bundled up libs, so that other wheels
>> could depend on them. OS-X does allow linking to relative paths, so this
>> should have been doable, but I never got anyone else to agree this was a
>> good idea, and I never found the roundtoits anyway. And it doesn't really
>> fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are
>> platform dependent and even worse, build-dependent.
>>
>> Meanwhile, conda was chugging along and getting a lot of momentum in the
>> Scientific community. And the core thing here is that conda was designed
>> from the ground up to support essentially anything, This means is supports
>> python packages that depend on non-python packages, but also supports
>> packages that have nothing to do with python (Perl, command line tools, what
>> have you...)
>>
>> So I have been focusing on conda lately.
>
>
> The whole reason I started this discussion is to make sure wheel has a
> standard way to do what is needed for those usecases.
>
> conda, rpm, deb, or eggs as used in enthought are all essentially the same:
> an archive with a bunch of metadata. The real issue is standardising on the
> exact formats. As you noticed, there is not much missing in the wheel *spec*
> to get most of what's needed. We've used eggs for that purpose for almost 10
> years at Enthought, and we did not need that many extensions on top of the
> egg format after all.
>
>
>>
>> Which brings me back to the question: should the python tools (i.e. wheel)
>> be extended to support more use-cases, specifically non-python dependencies?
>> Or do we just figure that that's a problem better solved by projects with a
>> larger scope (i.e. rpm, deb, conda, canopy).
>
>
> IMO, given that wheels do most of what's needed, it is worth supporting most
> simple usecases (compiled libraries required by well known extensions).
> Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to quite
> horrible custom hacks to support those cases.
>
> Hope that clarifies the intent,
>
> David

Then it sounds like I should read about the Enthought egg extensions.
It's something else than just defining a separate pypi name for "just
the libxml.so without the python bits"?