[Distutils] Multi-version import support for wheel files

[Nick Coghlan]

On 25 August 2013 21:02, Paul Moore <p.f.moore at gmail.com> wrote:
> Essentially, that is my question - and I'd like to know what Nick's proposal
> is here because I am not happy with the existing pkg_resources solution of
> using .pth files. I know there's a new feature being discussed on import-sig
> which may replace the need for pth files, but I don't want to see pth files
> being needed if that isn't available. Specific reasons I have for this:
>
> 1. The extra sys.path entries that get added (not for performance reasons,
> but because having a long sys.path is difficult for the interactive user to
> interpret when trying to see what's going on).
> 2. The whole "if I'm not in a site packages directory I won't work" issue
> 3. Questions of where things go on sys.path (if it depends on the order you
> "declare" things, it gets nasty when there are name clashes)

I'm not proposing copying over any of the implicit .pth file magic. As
far as I am aware, that's unrelated to the explicit multi-versioning
feature - I believe it's only needed if you want to install something
as an egg archive or subdirectory *and* have it available for import
by default. The namespace package related .pth files are also
unrelated.

Explicit multi-versioning is different - you just install the
directory, and you can't import from it by default. However,
pkg_resources can be used to activate it if it is needed to satisfy a
runtime requirement (or a dependency of a runtime requirement). If you
don't import pkg_resources, none of the parallel installed versions
will ever activate - you'll just get the version that is on sys.path
by default.

> The other thing I want to understand is how things would work if, for
> example, I wanted to use CherryPy 3 99% of the time, but occasionally needed
> CherryPy 2. Could I install CherryPy 3 as a normal (not multi-versioned)
> package, and then override it when needed? So only applications needing
> CherryPy 2 had to declare a version requirement?

Yep, this is exactly how Fedora & EPEL use it. CherryPy 3 is installed
as normal in site-packages (with an adjacent egg-info directory),
while CherryPy 2 is installed as an egg file containing the earlier
version of the module and the relevant metadata.

> More generally, how does a project choose whether to use run-time
> multi-versioning, vs metadata declaring a dependency for install time? Why
> would I ever write code that uses a run-time dependency on CherryPy 2,
> rather than saying in my metadata "requires: CherryPy = 2"? Or conversely
> why would I ever declare an install-time dependency rather than declaring my
> requirements at runtime "because that's more flexible"?

For Beaker, we use runtime dependencies because we build system
packages that run in the system Python, but need to run on RHEL 6
(CherryPy 2 as default) and also on recent versions of Fedora
(CherryPy 3 as default, CherryPy 2 available for multi-version
import). We also use it in our doc build scripts to handle the
different versions of Sphinx (Sphinx 1.x as default in Fedora, but
only available as a multi-version import on RHEL 6). The reason to
declare install time dependencies as well is that pkg_resources works
recursively - the dependencies of any distribution you activate will
be checked to ensure you're importing a consistent set of packages
into your application.

If you use virtual environments to create isolated stacks for each
application, these issues don't come up. On the other hand, if you're
trying to make thousands of packages play nice in a single Python
installation (as Linux distros do), then you need to deal with the
"parallel installation of mutually incompatible versions" problem. The
current Linux distro solution is to use pkg_resources, and having been
working with it for several months now, while I still think there are
some significant rough edges in the way the pkg_resources front end
works, I'm happy that the *file layout* the explicit multi-versioning
feature uses is a decent one that should translate to wheel files
fairly easily.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia