[Distutils] Mirroring PyPI JSON Locally

[Nick Coghlan]

On 22 August 2017 at 01:46, Wes Turner <wes.turner at gmail.com> wrote:
> ## Justify JSONLD
> - This is a graph. If we use an existing spec for graphs as JSON (ie
> JSONLD), we win:
>   - all of the tools that already exist for working with said graphs in that
> format
>   - easy indexability (as RDF quads)
>   - compatibility with compatible specs like ld-signatures

No, that's the argument I've already said doesn't work, since it
doesn't address the readability problem Paul mentioned: yes, JSON-LD
*can* represent aribtrary graphs, but to *read* a JSON-LD data
structure as a human, you need to know how *JSON-LD* represents
graphs.

I don't consider that an acceptable limitation: the raw metadata needs
to be readable by someone that's *only* familiar with the specifics of
dependency management in Python, and couldn't care less about the
representation of graphs as a general concept.

> ## Implement JSONLD
> - [ ] decide which URI(s) a project on {pypi,} is identified by
>    - some projects should not have an implicit pypi.org URI prefix
> - [ ] write a new JSONLD view for pypi and warehouse
> - [ ] write a JSONLD metadata spec for eggs and wheels

None of which are dependent on JSON-LD being the raw format for the
metadata - this can instead be done as a postprocessing step that
accepts any of the existing metadata formats as input. Defining such a
transformation is going to be critical for your goals of making the
JSON-LD representation useful anyway, as even if we defined a new
metadata format tomorrow, that would still mean there were more than
700 thousand existing releases on pypi.org that didn't natively
provide their metadata in that format.

The added bonus of doing things that way is that it means that you
*don't* need anyone else's agreement or consensus to start design work
- you can do an initial proof of concept using a domain you control,
similar to the way Donald started out by building the PyPI that he
wished existed as an independent service before we thanked him for his
efforts by lumbering him with the spectacularly difficult task of
figuring out how to upgrade or replace pypi.python.org itself :)

> ## Support metadata retrieval without exec'ing setup.py
>
> - develop a declarative format for expressing {sys.platform[...],}-dependent
> dependency edges

This is already part of PEP 508:
https://www.python.org/dev/peps/pep-0508/#environment-markers

This is why given a wheel file, you can *already* extract declarative
dependency metadata, using the METADATA file + PEP 508.

Given just an sdist, you can also do something similar by looking at
PKG-INFO, but that's less reliable (since that file may not even be
present, and even if it is, the sdist -> wheel build step may still
inject additional dependencies).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia