[Distutils] Handling the binary dependency management problem

[Nick Coghlan]

On 3 Dec 2013 00:19, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 2 December 2013 13:38, Tres Seaver <tseaver at palladion.com> wrote:
> > On 12/01/2013 06:38 PM, Paul Moore wrote:
> >> I understand that things are different in the Unix world, but to be
> >> blunt why should Windows users care?
> >
> > You're kidding, right?  90% or more of the reason for wheels in the
first
> > place is because Windows users can't build their own software from
> > source.  The amount of effort put in by non-Windows package owners to
> > support them dwarfs whatever is bothering you here.
>
> My point is that most of the complex binary compatibility problems
> seem to be Unix-related, and as you imply, Unix users don't seem to
> have much interest in using wheels except for local caching. So why
> build that complexity into the spec if the main users (Windows, and
> Unix users who won't ever publish wheels outside their own systems)
> don't have a need for it? Let's just stick with something simple that
> has limitations but works (practicality beats purity). My original
> bdist_simple proposal was a pure-Windows replacement for wininst.
> Daniel developed that into wheels which cater for non-Windows systems
> (I believe, precisely because he had an interest in the local cache
> use case). We're now seeing the complexities of the Unix world affect
> the design of wheels, and it's turning out to be a hard problem. All
> I'm trying to say is let's not give up on binary wheels for Windows,
> just because we have unsolved issues on Unix.

Huh? This is *exactly* what I am saying we should do - wheels *already*
work so long as they're self-contained.

They *don't* work (automatically) when they have an external dependency:
users have to obtain the external dependency by other means, and ensure
that everything is properly configured to find it, and that everything is
compatible with the retrieved version.

You're right that Christoph is doing two different things, though, so our
advice to him (or anyone that wanted to provide the cross-platform
equivalent of his current Windows-only stack) would be split:

- for all self-contained installers, also publish a wheel file on a custom
index server (although having a "builder" role on PyPI where project owners
can grant someone permission to upload binaries after the sdist is
published could be interesting)
- for those installers which actually form an integrated stack with shared
external binary dependencies, use the mechanisms provided by conda rather
than getting users to manage the external dependencies by hand (as
licensing permits, anyway)

>Whether solving the Unix
> issues is worth it is the Unix users' call - I'll help solve the
> issues, if they choose to, but I won't support abandoning the existing
> Windows solution just because it can't be extended to cater for Unix
> as well.

You appear to still be misunderstanding my proposal, as we're actually in
violent agreement. All that extra complexity you're worrying about is
precisely what I'm saying we should *leave out* of the wheel spec. In most
cases of accelerator and wrapper modules, the static linking and/or
bundling solutions will work fine, and that's the domain I believe we
should *deliberately* restrict wheels to, so we don't get distracted trying
to solve an incredibly hard external dependency management problem that we
don't actually need to solve at the wheel level, since anyone that actually
needs it solved can just bootstrap conda instead.

> I'm immensely grateful for the amount of work projects which are
> developed on Unix (and 3rd parties like Cristoph) put into supporting
> Windows. Far from dismissing that, I want to avoid making things any
> harder than they already are for such people - current wheels are no
> more complex to distribute than wininst installers, and I want to keep
> the impact on non-Windows projects at that level. If I come across as
> ungrateful, I apologise.

The only problem I want to explicitly declare out of scope for wheel files
is the one the wininst installers can't handle cleanly either: the subset
of Christoph's installers which need a shared external binary dependency,
and any other components in a similar situation.

Using wheels or native Windows installers can get you in trouble in that
case, since you may accidentally set up conflicts in your environment. The
solution is curation of a software stack built around that external
dependency (or dependencies), backed up by a packaging system that prevents
conflicts within a given local installation.

The mainstream Linux distros approach this problem by mapping everything to
platform-specific packages and trying to get parallel installation working
cleanly (a part of the problem I plan to work on improving post Python
3.4), but that approach doesn't scale well and is one of the factors
responsible for the notorious time lags between software being released on
PyPI and it being available in the Linux system package managers
(streamlining that conversion is one of my main goals for metadata 2.0).

The conda folks approach it differently by using hashes to lock in *exact*
versions of dependencies. From a compatibility management perspective, it's
functionally equivalent to bundling a complete stack into a single
monolithic installer, but has the benefit of still making it easy to only
install the pieces that you actually need locally.

Regards,
Nick.

>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131203/20374c4b/attachment.html>