[Distutils] distlib and wheel metadata

[Nick Coghlan]

On 15 February 2017 at 12:58, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, Feb 15, 2017 at 3:33 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> - "requires": list where entries are either a string containing a PEP
>> 508 dependency specifier or else a hash map contain a "requires" key
>> plus "extra" or "environment" fields as qualifiers
>> - "integrates": replacement for "meta_requires" that only allows
>> pinned dependencies (i.e. hash maps with "name" & "version" fields, or
>> direct URL references, rather than a general PEP 508 specifier as a
>> string)
>
> What's accomplished by separating these? I really think we should
> strive to have fewer more orthogonal concepts whenever possible...

It's mainly a matter of incorporating
https://caremad.io/posts/2013/07/setup-vs-requirement/ into the core
data model, as this distinction between abstract development
dependencies and concrete deployment dependencies is incredibly
important for any scenario that involves
publisher-redistributor-consumer chains, but is entirely non-obvious
to folks that are only familiar with the publisher-consumer case that
comes up during development-for-personal-and-open-source-use.

One particular area where this is problematic is in the widespread
advice "always pin your dependencies" which is usually presented
without the all important "for application or service deployment"
qualifier. As a first approximation:
pinning-for-app-or-service-deployment == good,
pinning-for-local-testing == good,
pinning-for-library-or-framework-publication-to-PyPI == bad.

pipenv borrows the Ruby solution to modeling this by having Pipfile
for abstract dependency declarations and Pipfile.lock for concrete
integration testing ones, so the idea here is to propagate that model
to pydist.json by separating the "requires" field with abstract
development dependencies from the "integrates" field with concrete
deployment dependencies.

In the vast majority of publication-to-PyPi cases people won't need
the "integrates" field, since what they're publishing on PyPI will
just be their abstract dependencies, and any warning against using
"==" will recommend using "~=" or ">=" instead. But there *are*
legitimate uses of pinning-for-publication (like the PyObjC
metapackage bundling all its subcomponents, or when building for
private deployment infastructure), so there needs to be a way to
represent "Yes, I'm pinning this dependency for publication, and I'm
aware of the significance of doing so"

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia