[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Chris Barker]

On Wed, May 20, 2015 at 12:57 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> This is why I'm such a big fan of richer upstream metadata with
> automated conversion to downstream formats as my preferred long term
> solution - this isn't a "pip vs conda" story, it's "pip vs conda vs
> yum vs apt vs MSI vs nix vs zypper vs zc.buildout vs enstaller vs PyPM
> vs ....".


hopefully not "versus", but "working with" ;-) -- but very good point. If
python can do things to make it easier for all these broader systems,
that's a "good thing"


> The main differences I see with conda relative to the other downstream
> package management systems is that it happened to be made by folks
> that are also heavily involved in development of Python based data
> analysis tools,


Which is to say Python itself.


> and that some of its proponents want it to be the "one
> package management tool to rule them all".


I don't know about that -- though another key point is that it is cross
platform (platform independent) -- it may be the only one that does that
part well.


> I consider the latter
> proposal to be as outlandish an idea as believing the world only needs
> one programming language - just as with programming languages,
> packaging system design involves making trade-offs between different
> priorities, so you can't optimise for everything at once. conda's an
> excellent cross-platform end user focused dependency management
> system. This is a good thing, but it does mean conda isn't a suitable
> candidate for use as an input format for other tools that compete with
> it.
>

Hmm -- that's true. But it is, as you said " cross-platform end user
focused dependency management system" that handles python well, in addition
to other things, including libs python may depend on.

As such, it _could_ play the role that pip+wheel (secondarily pypi) play in
the python ecosystem. You'd still need something like distutils and/or
setuptools to actually handle the building, etc.

And IF we wanted the "official" package manager for python to fully support
dynamic libs, etc, as well  as non-python associated software, then it
would make sense to use conda, rather than keep growing pip_wheel until it
duplicated conda's functionality.

But I don't get the impression that that is an end-goal for PyPa, and I'm
not sure it should be.

As far as the "we could use a better dynamic linking story for Windows
> and Mac OS X" story goes, now that I understand the general *nix case
> is considered out of scope for the situations Chris is interested in,
>

exactly, -- just like it linux is out of scope for compiled wheels

I think there's a reasonable case to be made for being able to
> *bundle* redistributable dynamically linked libraries with a wheel
> file, and for the build process of *other* wheel files to be able to
> rely on those bundled external libraries.


yup -- that's what I have in mind.


> I originally thought the
> request was about being able to *describe* the external dependencies
> in sufficient detail that the general case on *nix could be handled,
> or that an appropriate Windows or Mac OS X binary could be obtained
> out of band, rather than by being bundled with the relevant wheel
> file.
>

Sure would be nice, but no, -- I have no fantasies about that.

Getting a bundling based model to work reliably is still going to be
> difficult (and definitely more complicated than static linking in
> cases where data sharing isn't needed), but it's not intractable the
> way the general case is.
>

Glad you agree -- so the rabbit hole may not be that deep?

There isn't much that should change in pip+wheel+metadata to enable this.
So the way to proceed, if someone wants to do it, could be to simply hack
together some binary wheels of a common dependency or two, build wheels for
a package or two that depend on those, and see how it works.

I dont know if/when I'll find the roundtoits to do that -- but I have some
more detailed ideas if anyone wants to talk about it.

Then it becomes a social issue -- package maintainers would have to
actually use these new sharedlib wheels to build against. But that isn't
really that different than the current case of deciding whether to include
a copy of a dependent python package in your distribution -- and one we
made it easy for users to get dependencies, folks have been happy to shift
that burden elsewhere.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150520/ea1ea057/attachment.html>