[Distutils] Handling the binary dependency management problem

[Paul Moore]

On 2 December 2013 14:19, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
> Basically if
> 1) There is at least one single consistent set of built wheels for
> Windows/OSX for any popular set of binary-interdependent packages.
> 2) A way to automatically detect incompatibilities and to
> automatically find compatible built wheels.
> then *a lot* of packaging problems have been solved.
>
> Part 1 already exists. There are multiple consistent sets of built
> installers (not wheels yet) for many hard to build packages. Part 2
> requires at least some changes in pip/PyPI.

Precisely.

But isn't part 2 at least sort-of solved by users manually pointing at
"the right" index? The only files on PyPI are compatible with each
other and externally hosted files (thanks for the pointer to the
matplotlib binaries, BTW) won't get picked up automatically by pip so
users have to set up their own index (possibly converting
wininst->wheel) and so can manually manage the compatibility process
if they are careful.

If people start uploading incompatible binaries to PyPI, I expect a
rash of bug reports followed very quickly by people settling down to a
community-agreed standard (in fact, that's probably already happened).
Incompatible builds will remain on external hosts like Cristoph's.

It's not perfect, certainly, but it's no worse than currently.

For any sort of better solution to part 2, you need *installed
metadata* recording the ABI / shared library details for the installed
files. So this is a Metadata 2.0 question, and not a compatibility tag
/ wheel issue (except that when Metadata 2.0 gets such information,
Wheel 2.0 probably needs to be specified to validate against it or
something). And on that note, I agree with Nick that we don't want to
be going there at the moment, if ever. I just disagree with what I
thought he was saying, that we should be so quick to direct people to
conda (at some point we could debate why conda rather than ActiveState
or Enthought, but tbh I really don't care...) I'd go with something
along the lines of:

"""
Wheels don't attempt to solve the issue of one package depending on
another one that has been built with specific options/compilers, or
links to specific external libraries. The binaries on PyPI should
always be compatible with each other (although nothing checks this,
it's simply a matter of community standardisation), but if you use
distributions hosted outside of PyPI or build your own, you need to
manage such compatibility yourself. Most of the time, outside of
specialised areas, it should not be an issue[1].

If you want guaranteed compatibility, you should use a distribution
that validates and guarantees compatibility of all hosted files. This
might be your platform package manager (apt or RPM) or a bundled
Python distribution like Enthought, Conda or Activestate.
"""

[1] That statement is based on *my* experience. If problems are
sufficiently widespread, we can tone it down, but let's not reach the
point of FUD.

Paul