[Distutils] What metadata does pip actually need about sdists?

[Nathaniel Smith]

On Sun, Oct 11, 2015 at 10:12 PM, Marcus Smith <qwcode at gmail.com> wrote:
>>
>> it's a fact of life that the same
>> source release may be configured in different ways that create
>> different resulting dependencies. NumPy is one example of this, but
>> it's hardly unusual
>
>
> I've tried to respond to this point twice.. but let me try again. :  )
> If I'm confused, please help me.
>
> aren't the "different resulting dependencies" non-python, non-pypi system
> dependencies?
>
> the dependencies the sdist would cover are the python/pypi dependencies, and
> they don't usually vary based on building extensions
>
> as for representing non-python distro dependencies and build variances, I
> mentioned before that PEP426 extensions might be used to cover that, but
> that's still an open question I think..

Ah, sorry, there was some discussion of this buried deep inside the
giant thread :-)

For NumPy/SciPy/scikit-learn/etc. we have the problem that we need to
somehow distribute the standard linear algebra libraries like BLAS and
LAPACK. For e.g. debian packages, or a self-built version on Linux,
these are generally treated as distro dependencies -- the user is
responsible for making sure that they're (somehow) installed, and then
we link to them directly. For wheels, though, we can't assume this --
not just because there's no spec for distro dependencies, but because
even if there would then it'd probably be fragile (no guarantee that
different distros have compatible ABIs) and it certainly wouldn't work
on Windows or OS X. So instead, the current plan is that we're going
to drop the libraries inside a wheel and upload it to PyPI:
  https://mail.python.org/pipermail/distutils-sig/2015-October/027164.html
and our wheels on pypi will have an install_requires on this special
library wheel.

Compared to PEP426-extensions idea, this has the advantages that (a)
it doesn't require a PEP, (b) there's no question about whether it
will work :-).

Another example of where this might arise is if a package [1] wants to
optionally use dynd as its array backend instead of numpy [2], so it
would need to optionally link to the dynd C API instead of or in
addition to the numpy API. But dynd is a young and rapidly changing
project, so you might want to support this as an experimental build
configuration while still uploading "official" wheels that depend on
numpy only. But the configuration that does link to dynd would still
want to fetch dynd using python-level dependencies [3].

Does that help?

-n

[1] This isn't terribly hypothetical, pandas is one of the most
popular data processing libraries in the world and they have an
experimental branch doing exactly this:
https://github.com/pydata/pandas/issues/8643
[2] https://github.com/libdynd/libdynd
[3] https://pypi.python.org/pypi/dynd

-- 
Nathaniel J. Smith -- http://vorpus.org