[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

Seems like you could extend wheel to do that easily.
On Apr 13, 2015 4:19 PM, "David Cournapeau" <cournape at gmail.com> wrote:

> I would advise against using or even reading about our egg extensions, as
> the implementation is full of legacy (we've been doing this many years :)
> ): http://enstaller.readthedocs.org/en/master/reference/egg_format.html
>
> This is what we use on top of setuptools egg:
>
>  - ability to add dependencies which are not python packages (I think most
> of it is already handled in metadata 2.0/PEP 426, but I would have to
> re-read the PEP carefully).
>  - ability to run post/pre install/remove scripts
>  - support for all the of the autotools directories, with "sensible"
> mapping on windows
>  - a few extensions to the actual binary format (adding support for
> symlinks is the only one I can think of ATM).
>
> Everything else is legacy you really don't want to know (see here if you
> still want to
> http://enstaller.readthedocs.org/en/master/reference/egg_format.html)
>
> David
>
> On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>> On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau <cournape at gmail.com>
>> wrote:
>> >
>> >
>> > On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker <chris.barker at noaa.gov>
>> > wrote:
>> >>
>> >> NOTE: I don't work for any of the companies involved -- just a somewhat
>> >> frustrated user... And someone that has been trying for years to make
>> things
>> >> easier for OS-X users.
>> >>
>> >>>> I’m not sure what (3) means exactly. What is a “normal” Python, do
>> you
>> >>>> modify Python in a way that breaks the ABI but which isn’t reflected
>> in the
>> >>>> standard ABI tag?
>> >>>
>> >>>
>> >>> It could be multiple things. The most obvious one is that generally.
>> >>> cross-platforms python distributions will try to be "relocatable"
>> (i.e. the
>> >>> whole installation can be moved and still work). This means they
>> require
>> >>> python itself to be built a special way. Strictly speaking, it is not
>> an ABI
>> >>> issue, but the result is the same though: you can't use libraries from
>> >>> anaconda or canopy on top of a normal python
>> >>
>> >>
>> >> But why not? -- at least for Anaconda, it's because those libraries
>> likely
>> >> have non-python dependencies, which are expected to be installed in a
>> >> particular way. And really, this is not particular to Anaconda/Canopy
>> at
>> >> all. Python itself has no answer for this issue, and eggs and wheels
>> don't
>> >> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in
>> order to
>> >> build a binary wheel with non-python dependencies (let's say something
>> like
>> >> libjpeg, for instance), you need to either:
>> >>  - assume that libjpeg is installed in a "standard" place -- really no
>> >> solution at all (at least outside of linux)
>> >>  - statically link it
>> >>  - ship the dynamic lib with the package
>> >>
>> >> For the most part, the accepted solution for OS-X has been to
>> statically
>> >> link, but:
>> >>
>> >>  - it's a pain to do. The gnu toolchain really likes to use dynamic
>> >> linking, and building a static lib that will run on a
>> >> maybe-older-than-the-build-system machine is pretty tricky.
>> >>
>> >>  - now we end up with multiple copies of the same lib in the python
>> >> install. There are a handful of libs that are used a LOT. Maybe there
>> is no
>> >> real downside -- disk space and memory are cheap these days, but it
>> sure
>> >> feels ugly. And I have yet to feel comfortable with having multiple
>> versions
>> >> of the same lib linked into one python instance -- I can't say I've
>> seen a
>> >> problem, but it makes me nervous.
>> >>
>> >> On Windows, the choices are the same, except that: It is so much
>> harder to
>> >> build many of the "standard" open source libs that package authors are
>> more
>> >> likely to do it for folks, and you do get the occasional "dll hell"
>> issues.
>> >>
>> >> I had a plan to make some binary wheels for OS-X that were not really
>> >> python packages, but actually just bundled up libs, so that other
>> wheels
>> >> could depend on them. OS-X does allow linking to relative paths, so
>> this
>> >> should have been doable, but I never got anyone else to agree this was
>> a
>> >> good idea, and I never found the roundtoits anyway. And it doesn't
>> really
>> >> fit into the PyPi, pip, wheel, etc. philosphy to have dependencies
>> that are
>> >> platform dependent and even worse, build-dependent.
>> >>
>> >> Meanwhile, conda was chugging along and getting a lot of momentum in
>> the
>> >> Scientific community. And the core thing here is that conda was
>> designed
>> >> from the ground up to support essentially anything, This means is
>> supports
>> >> python packages that depend on non-python packages, but also supports
>> >> packages that have nothing to do with python (Perl, command line
>> tools, what
>> >> have you...)
>> >>
>> >> So I have been focusing on conda lately.
>> >
>> >
>> > The whole reason I started this discussion is to make sure wheel has a
>> > standard way to do what is needed for those usecases.
>> >
>> > conda, rpm, deb, or eggs as used in enthought are all essentially the
>> same:
>> > an archive with a bunch of metadata. The real issue is standardising on
>> the
>> > exact formats. As you noticed, there is not much missing in the wheel
>> *spec*
>> > to get most of what's needed. We've used eggs for that purpose for
>> almost 10
>> > years at Enthought, and we did not need that many extensions on top of
>> the
>> > egg format after all.
>> >
>> >
>> >>
>> >> Which brings me back to the question: should the python tools (i.e.
>> wheel)
>> >> be extended to support more use-cases, specifically non-python
>> dependencies?
>> >> Or do we just figure that that's a problem better solved by projects
>> with a
>> >> larger scope (i.e. rpm, deb, conda, canopy).
>> >
>> >
>> > IMO, given that wheels do most of what's needed, it is worth supporting
>> most
>> > simple usecases (compiled libraries required by well known extensions).
>> > Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to
>> quite
>> > horrible custom hacks to support those cases.
>> >
>> > Hope that clarifies the intent,
>> >
>> > David
>>
>> Then it sounds like I should read about the Enthought egg extensions.
>> It's something else than just defining a separate pypi name for "just
>> the libxml.so without the python bits"?
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150413/bb8650de/attachment.html>