[Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)

[Nick Coghlan]

On 21 Jun 2013 07:40, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 20 June 2013 21:54, Daniel Holth <dholth at gmail.com> wrote:
>>
>> These metadata PEPs are
>> mostly about the installer, database of installed files, and package
>> indexes.
>
>
> In essence (and I freely admit I don't fully understand the details yet
myself) I think the idea is that PEPs 426 and 440 define a set of data that
can be published by packages. The *how* of publishing that data is still to
be defined - the idea is that there will be the appropriate metadata JSON
files available, but no tools currently produce that format. Equally,
installers like pip will be able to read the metadata and ensure that
dependencies are installed appropriately (for example) but again, that will
be a future change to pip.
>
> For now, the PEPs define a standard for tools to *use*, but leave the
creation of such tools (or modification of existing tools) as a later step.
>
> I don't think the PEPs say anything about a plan for implementing the
spec (although I haven't read the PEPs in detail yet, so I may have missed
something)

Right, the general implementation plan isn't part of the PEP, aside from
the big note at the beginning about all the *other* PEPs that will be
needed to finish bootstrapping this thing.

> but my assumption is that it'll go something like this:
>
> 1. Python setup.py sdist and python setup.py dist_info will be changed to
generate pymeta.json files. But that will be for Python 3.4 only (there's a
big problem if this doesn't make it into 3.4...). Unless there's a
distutils backport or similar.

The initial vehicles for migration will be Daniel's bdist_wheel command and
upgrades to setuptools.

Standard library support may not land for 3.4, but we will at least have
the pip bootstrapping mechanism in place (which in turn will allow
retrieval of setuptools).

Until the updated packaging ecosystem stabilises, adopting the standard
library's update rate would be a bad idea.

Besides, a solution that doesn't work back as far Python 2.6 faces
significant barriers to community adoption.

> 2. Pypi will be changed to publish metadata from pymeta.json, when it is
present, which tools like pip can consume (see later).

/me lets cat out of bag

Donald's actually working on a second generation PyPI design that uses PEP
426 as its underlying data model for distributions. It will probably
include the capability to derive 2.0 metadata from legacy metadata, so
installers can benefit from the new mechanisms at install time, even for
existing distributions. Many of the security constraints and backwards
compatibility tweaks in the PEP actually come from that design work.

With the spec now stabilised, we'll likely flip the public bit on that repo
soon, and start working on some concrete plans to spin up a test instance
alongside the existing PyPI service :)

Along with Daniel ensuring that bdist_wheel can emit the format, Donald's
work on the server side has been a key part of keeping the metadata spec
grounded in reality.

Once we flip the public bit on the repo, the next generation PyPI will
serve alongside Daniel's existing wheel project as the reference
implementations for the spec.

> 3. Wheel will put pymeta.json into wheels, from the pymeta.json created
by (1) and likely by converting legacy egg-info metadata as a fallback.
Wheel will also write pymeta.json on install.

On install, it should be covered by the existing "copy the dist-info
contents from the wheel" behaviour, but that's the general idea.

sdist 2.0 will just deprecate PKG-INFO and replace it with a self
identifying wheel-style dist-info subdirectory.

> 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install,
and use installed pymeta.json (again probably falling back to legacy
formats where pymeta.json isn't present).
> 5. Distlib will follow suit.

Yup (although *definitely* falling back to legacy formats).

>
> The big gap here is older versions of Python. Maybe a setuptools plugin
to generate the new metadata in sdists would be viable. Maybe package
writers could provide pymeta.json by hand, although I seriously doubt many
will bother.
>
> How close am I to reality here?

Close, except that this is explicitly an "ecosystem first" plan, with the
standard library playing catch up later. That's the big reason I sought
(and received) agreement at this year's language summit for us to discuss
and approve packaging related PEPs right here on distutils-sig rather than
having to repost them to python-dev for ratification.

The only "must have" on the list for 3.4 is the pip bootstrapping, and
Richard and I hope to get that squared away at the PyCon AU sprints next
month.

Cheers,
Nick.

>
> Paul
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130621/78ed9efb/attachment-0001.html>