[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Paul Moore]

On 12 May 2014 20:58, M.-A. Lemburg <mal at egenix.com> wrote:
> If it helps convince you that allowing verifiable external
> links per default is a good thing for user experience, we will
> register the distribution file download URLs with the PyPI
> web API.

Personally, I'm on the fence over that one. There are so few projects
with verifiable external links that it's hard to be sure where the
costs and gains are. Certainly if more projects use the deature that
will affect the decision, but I'm not going to be the one to say how
much weight individual projects carry...

To be clear, --allow-external and --allow-all-external exist right
now. That matches PEP 438. Either of making --allow-all-external the
default or treating both verifiable and unverifiable external links as
opt-in under the same option would be changes to both the PEP and pip,
and neither has happened yet.

> You can think of it as per-package PyPI-style simple index
> that's registered with PyPI in a secure way (via the check sum
> hash).
>
> There's most certainly room for improvement and I'm open for
> discussions.

That sounds like an interesting proposal, and worth discussing. I
presume it would need a PEP to implement, as there would be impacts on
pip, PyPI and warehouse at a minimum, and possibly easy_install and at
some point distil/distlib.

>> I'm not quite sure how you expect this will work, but it's probably
>> important that you get involved with the various packaging PEPs. The
>> only way I can see such a solution working with pip would be if you
>> have a customised setup.py.
>
> Yep, since that's the way installers (not only pip) work when
> they see a source distribution.

OK. The move away from having executable code run at install time is
well outside the scope of this debate or any discussion around pip at
the current point. You might want to look for threads mentioning the
"meta build system" for current thinking, but it's very much
vapourware at the moment.

> Binary installs are nice, but they are not the answer to everything
> and no matter how much meta data you put into static files,
> there will always be cases where that meta data description just
> doesn't include those bits you would need to complete the setup,
> esp. for packages with C extensions and more complex external
> dependencies or setup requirements. (*)
>
> The setup.py interface makes all this possible, which is why so
> many Python packages use it to configure themselves automatically.

See my response to Stefan, and those threads about the meta build
system I mentioned earlier.

> Deprecating this interface would make some distributions impossible
> to install without manual user intervention and we'd be back to the
> Makefile.pre.in days.

Nothing is being designed in private, but any discussions that do
happen will be on distutils-sig, so I'd recommend monitoring that list
so you don't get taken by surprise.

Paul