[Wheel-builders] pinned-down dependencies for building wheel versus loose dependencies for experimentation

[Philippe Ombredanne]

On Thu, Sep 8, 2016 at 11:21 PM, Vitaly Kruglikov
<vkruglikov at numenta.com> wrote:
> My colleagues and I are struggling to reconcile this problem: if our CI
> system only tests our library with a specific set of pinned-down
> dependencies, how can we deploy a wheel to PyPi that references a looser
> set of dependencies? What guarantee would there be that it will work
> correctly with the looser set of dependencies, since we can¹t possibly
> test with **all** of their combinations?
>
> The suggestion from your previous post for solving my problem by deploying
> library A with loose dependencies to PyPy as well as another ³codeless²
> library B with only setup.py containing the pinned-down dependencies would
> likely work. However, the additional complexity of the solution (more
> moving parts) is telling me that the problem I am trying to solve may be
> going against the grain of what python packaging and deployment intended.

You can do this very simply IMHO by having a regular setup.py with
flexible deps; and a requirements.txt with pinned deps and a
setup_pinned.py that includes no code and sources your pinned
requirements.txt. And you publish both wheels to Pypi. quite simple
IMHO

> If it¹s common practice for wheels deployed to PyPi to incorporate
> loosely-versioned dependencies, then how do developers mitigate the risk
> that their wheel might fail with some combination of those dependencies?

This is not a matter of common practice, this is the intended usage
and the sane way  to deal with two contexts:
- as a library or framework developer, I shall give flexibility to my
users with their deps
- as an application developer, I want to have something that works
with a set of pinned deps that I have tested

In your case, you want to fill in both usage in one place... hence
your challenge: this cannot be sanely handled with a single
description of deps: you need both flexible and pinned.

I have similar requirements for one of my projects which is both a
framework/library and an application.
I solve this this way which works for me:
- my setup.py has flexible version ranges, mostly with a lower version
limit to support use as a lib
- when used as an app, the distribution contains the whole recursive
set (vendored) of pinned dependent wheels and I use a small
configuration script that pip installs only from these pinned deps.

The benefits to me are:
1. when used from Pypi as a lib in some other project, my package has
enough flexibility to avoid major conflicts with other package
requirements
2. when used as an app, I have total control of which wheel and which
file is installed exactly
3. when used as an app, the app is fully self-contained and does not
depend on Pypi or network availability

This is surely not the only to do this: the suggested approach with
two wheels (one flexible , one pinned) is another way.
And there are surely other ways to handle this. But the added
complexity is likely always needed if you want to support both a
flexible and a pinned set of deps.

-- 
Cordially
Philippe Ombredanne