[Distutils] distlib and wheel metadata

[Freddy Rietdijk]

> Maybe it would help if you have a concrete example of a
scenario where they would benefit from having this distinction?

In the Nix package manager (source distribution with binary substitutes)
and Nixpkgs package set we typically require the filename and hash of a
package. In our expressions we typically pass an url (that includes the
name), and the hash. The url is only needed when the file isn't in our
store. This is convenient, because if an url is optional this allows you to
pre-fetch or work with mirrors. All we care about is that we get the file,
not how it is provided. This applies for source archives, but behind the
scenes also for binary substitutes. With Nix, functions build a package,
and dependencies are passed as function arguments with names that
typically, but not necessarily, resemble the dependency name.

Now, a function that builds a package, a package builder, only needs to be
provided with abstract dependencies; it just needs to know what it should
look for, "we need 'a' numpy, 'a' scipy, 'a compiler that has a certain
interface and can do this job'", etc.. Version numbers can help in order to
fail prematurely, but generally only bounds, not a pinned value. Its up to
another tool to provide the builder with the actual packages, the concrete
dependencies to the builder. And this tool might fetch it from PyPI, or
from GitHub, or...

The same goes for building, distributing and installing Python packages.
Setuptools shouldn't bother with versions (except the constraints in case
of libraries) or wherever a source comes from but just build or fail. Pip
should just fetch/resolve and pass concrete dependencies to whatever
builder (Setuptools, Flit), or whatever environment (virtualenv) needs it.

It's quite frustrating as a downstream having to deal with packages where
versions are pinned unnecessarily and therefore I've also requested on the
Setuptools tracker a flag that ignores constraints [1] (though I fear I
would have to pull up my sleeves for this one :) ) .

[1] https://github.com/pypa/setuptools/issues/894

On Wed, Feb 15, 2017 at 3:11 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Feb 15, 2017 at 5:27 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 15 February 2017 at 12:58, Nathaniel Smith <njs at pobox.com> wrote:
> >> On Wed, Feb 15, 2017 at 3:33 AM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >>> - "requires": list where entries are either a string containing a PEP
> >>> 508 dependency specifier or else a hash map contain a "requires" key
> >>> plus "extra" or "environment" fields as qualifiers
> >>> - "integrates": replacement for "meta_requires" that only allows
> >>> pinned dependencies (i.e. hash maps with "name" & "version" fields, or
> >>> direct URL references, rather than a general PEP 508 specifier as a
> >>> string)
> >>
> >> What's accomplished by separating these? I really think we should
> >> strive to have fewer more orthogonal concepts whenever possible...
> >
> > It's mainly a matter of incorporating
> > https://caremad.io/posts/2013/07/setup-vs-requirement/ into the core
> > data model, as this distinction between abstract development
> > dependencies and concrete deployment dependencies is incredibly
> > important for any scenario that involves
> > publisher-redistributor-consumer chains, but is entirely non-obvious
> > to folks that are only familiar with the publisher-consumer case that
> > comes up during development-for-personal-and-open-source-use.
>
> Maybe I'm just being dense but, umm. I don't know what any of these
> words mean :-). I'm not unfamiliar with redistributors; part of my
> confusion is that this is a concept that AFAIK distro package systems
> don't have. Maybe it would help if you have a concrete example of a
> scenario where they would benefit from having this distinction?
>
> > One particular area where this is problematic is in the widespread
> > advice "always pin your dependencies" which is usually presented
> > without the all important "for application or service deployment"
> > qualifier. As a first approximation:
> > pinning-for-app-or-service-deployment == good,
> > pinning-for-local-testing == good,
> > pinning-for-library-or-framework-publication-to-PyPI == bad.
> >
> > pipenv borrows the Ruby solution to modeling this by having Pipfile
> > for abstract dependency declarations and Pipfile.lock for concrete
> > integration testing ones, so the idea here is to propagate that model
> > to pydist.json by separating the "requires" field with abstract
> > development dependencies from the "integrates" field with concrete
> > deployment dependencies.
>
> What's the benefit of putting this in pydist.json? I feel like for the
> usual deployment cases (a) going straight from Pipfile.lock -> venv is
> pretty much sufficient, with no need to put this into a package, but
> (b) if you really do want to put it into a package, then the natural
> approach would be to make an empty wheel like
> "my-django-app-deploy.whl" whose dependencies were the contents of
> Pipfile.lock.
>
> There's certainly a distinction to be made between the abstract
> dependencies and the exact locked dependencies, but to me the natural
> way to model that distinction is by re-using the distinction we
> already have been source packages and binary packages. The build
> process for this placeholder wheel is to "compile down" the abstract
> dependencies into concrete dependencies, and the resulting wheel
> encodes the result of this compilation. Again, no new concepts needed.
>
> > In the vast majority of publication-to-PyPi cases people won't need
> > the "integrates" field, since what they're publishing on PyPI will
> > just be their abstract dependencies, and any warning against using
> > "==" will recommend using "~=" or ">=" instead. But there *are*
> > legitimate uses of pinning-for-publication (like the PyObjC
> > metapackage bundling all its subcomponents, or when building for
> > private deployment infastructure), so there needs to be a way to
> > represent "Yes, I'm pinning this dependency for publication, and I'm
> > aware of the significance of doing so"
>
> Why can't PyObjC just use regular dependencies? That's what distro
> metapackages have done for decades, right?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170215/a38f252c/attachment.html>