[Distutils] What metadata does pip actually need about sdists?

[Nathaniel Smith]

On Sun, Oct 11, 2015 at 10:48 AM, Marcus Smith <qwcode at gmail.com> wrote:
>
>>
>> 2) after unpacking this sdist it then calls 'setup.py egg_info' to get
>> the full metadata for the wheel
>
>
> I wouldn't say "get the full metadata for the wheel".  it's not a wheel yet.
> `egg_info` run so we can use the pkg_resources api to find the dependencies.
>
>
>>
>> Specifically what it does with this is
>> extract the setup_requires and install_requires fields
>
>
> specifically, we call `requires` on pkg_resources distribution objects
> https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583
>
>
>>
>> 3) eventually it actually builds the package, and this produces a
>> wheel (or wheel equivalent) that has its own metadata (which often
>> matches the metadata from egg_info in step (2), but not always)
>
>
> "not always"?  not following your point
> they're 2 different formats, but should contain the same essential
> information.
> here's the wheel code that does the conversion
> https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default&fileviewer=file-view-default#metadata.py-90

I just meant that due to the fact that you're running two chunks of
arbitrary code, there's no way to statically guarantee that the
metadata produced in the two calls to 'setup.py' is going to match. In
the message here:
  https://mail.python.org/pipermail/distutils-sig/2015-October/027161.html
I linked to some packages whose setup.py's actually report different
dependency information in the egg_info and build phases.

I'm not saying this is a good or bad or even particularly relevant
thing, just observing that it's a fact about how the current system
works :-).

(And this is part of why my draft PEP that set all this off just has a
single atomic "build a wheel" operation instead of splitting this into
two phases.)

>
>
>>
>> name and version before it runs egg_info, I assume that what this
>> means is that it's crucial for pip to have static access to dependency
>> information?
>
>
> yes
>
>
>>
>> It would be somewhat convenient if sdists did list their binary
>> dependencies:
>
>
> not sure about your insertion of "binary" here.
> pip is concerned with finding python project dependencies (i.e. name and
> version constraints) in the sdist
> and then based on the current install environment, it will further constrain
> the wheels chosen based on architecture and python implementation.
> and to be perfectly clear, none of this deals with non-python OS/Distro
> requirements.

I didn't mean anything in particular by "binary", sorry if that threw
you off. It is probably because I am still finding it useful to think
of source packages and binary packages as logically distinct, i.e., a
source package is a black box that can produce a binary package and
when I have a dependency it names a binary package that I want (which
can be obtained either by downloading it or by finding a source
package that will produce that binary package and building it).

>> 3) if any of the packages-to-be-installed are sdists, then fetch them,
>> run egg_info or build them or whatever to get their real dependencies,
>> add these to the graph, and go to step 1
>
>
> this is the pain we don't want in the future.
>
> since we're simply talking about name/version constraints (not
> platform/python), It's hard to conceive that we'd agree on an sdist spec
> that didn't include that.
>
>
>>
>> Do the relevant pip maintainers
>> even read this mailing list? :-)
>
>
> I try  :  )

Sorry, I realized that may have come across as much more negative than
I intended :-). I just realized at the end of typing this whole thing
that I wasn't even sure if I was sending it to the right place :-).

-- 
Nathaniel J. Smith -- http://vorpus.org