[Distutils] Metadata extension discovery?

[C Anthony Risinger]

On Fri, Sep 12, 2014 at 10:25 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> On 13 Sep 2014 00:20, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> >
> > Yes, it sounds like things are getting complex here and I'm not sure I
> > follow why. At the moment, the metadata for a distribution is
> > generated when setup.py is run, and is stored in the wheel and in the
> > installed dist-info directory when the distribution is installed.
> >
> > The proposal here seems to be that something *else* could add metadata
> > to a distribution, from outside of that distribution.
>
> Correct - for example, a build tool might need to record additional
> compatibility constraints in a built wheel file.
>
> Software distribution is a pipeline, rather than a "once and done" thing,
> so we need to keep that in mind as we design the metadata.
>
> There's at least four phases:
>
> - pre-archiving metadata (not currently specified, setup.py/setuptools as
> the de facto standard for the time being)
> - archive metadata (metadata 2.0)
> - build metadata (wheel 2.0?)
> - install metadata (install DB 2.0?)
>
> Redistributors also need a way to inject our metadata additions (like the
> proposed "python.integrator" extension in PEP 459, which I may rename to
> "python.redistributor")
>
> The unordered-by-default nature of JSON makes it a difficult format to
> reliably patch (even in "append only" mode), so providing orthogonal files
> becomes a much easier option.
>
> Now, if we're going to have orthogonal files, does it make more sense to
> organise them by phase of distribution, or by the extension name? I suspect
> organising by phase does make more sense, but see enough merit in
> organising by extension to at least consider the idea.
>
> Cheers,
> Nick.
>
> P.S. Note that Daniel's old idea of a persistent on disk SQLite metadata
> cache, as well as updating the import hook system to abstract away the
> publication of packaging metadata are also both still open design questions.
>

I'm glad this came up because I had been wondering for some time how this
would be handled.

While I have a couple other use cases related to a special
installer-generator we use (opting-in to __future__'s, among others) I am
specifically curious how pydist.json will handle 'namespace packages'...

eg. I create package `foo.bar`, and from a packaging perspective it's an
independent thing, but functionally, it is an extension of `foo`, and
*parts* of it's metadata may very well apply to the `foo` package as a
whole.

This wasn't meant to be an [ANN], but my tool:

https://github.com/xtfxme/zippy

...uses metadata 2.0 exclusively and is capable of:

 - compiling *any* python C-extension statically (numpy, whatever)
 - extends `-m` to support `-m package.module:callable` (NICE)
 - allows forwarding argv[0] != 'python*' to python.zip/bin/{argv[0]}
 - (many others)

...not the best examples, but for such things I'd like to allow an opt-in
of sorts by the system integrator (note, not related to packager), so they
can choose which features to apply to which packages at build-time... the
current way I am considering is via `pydist.d` because, as Nick stated, it
is familiar and makes sense to most people.

Couple questions:

 - would the extension drop `.d` fragments into it's own dist-info, or the
parent package?
 - if the latter, how does this work in the face of `provides` and
`replaces` and such?

Namespace packages I think are the clearest concept for why this feature is
needed.

-- 

C Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140912/eadc3e2e/attachment.html>