[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Paul Moore]

On 12 May 2014 13:28, M.-A. Lemburg <mal at egenix.com> wrote:
>> So, some questions:
>>
>> 1. Is MAL aware that egenix-mx-base is not verifiably externally
>> hosted[1], and if so, what is he asking for? Automatic download with
>> no need for opt-in of unverifiable external downloads? That seems
>> pretty much in conflict with the whole intent of PEP 438.
>
> What we are implementing is a proposal that I brought up before
> PEP 438 was put in place:
>
> Instead of linking directly to all packages, we put up a verifiable
> link to an index page with verifiable links, with the net effect
> being that tools can verify the whole chain.

Thanks for clarifying. My main concern is that people do actually
understand the requirements for "safe" external hosting (I discovered
that I certainly didn't - it's quite complex!). From what you say, you
do understand the requirements but have chosen not to follow them at
this time. That's fine, I'm not trying to debate the validity of your
choice. But in the context of PEP 438, that means that users of the
egenix downloads will have to opt into unverifiable downloads in order
to install using pip. We're working towards improving the user
experience for end users who need to install unverifiable downloads -
I'd expect that one consequence of this will be that we'll be better
able to communicate the situation to those users, which will reduce
the confusion.

I don't know if you're suggesting that your proposal is still
something that we should be looking at implementing. I'm happy to
discuss that, but it should probably be a separate thread.

> since shifted focus to working on a web installer which solves
> multiple problems at once:
[...]
> With the web installer, we'd just have to upload one file
> per release.

I'm not quite sure how you expect this will work, but it's probably
important that you get involved with the various packaging PEPs. The
only way I can see such a solution working with pip would be if you
have a customised setup.py. As the general trend is towards binary
installs using wheels, with pip eventually deprecating sdist installs
and running setup.py directly, we will likely miss your use case
unless you get involved in those discussions (they are on the back
burner at the moment, but will likely resurface at some point -
there's currently a work-in-progress PR for pip to use a "wheel
cache", where the normal install route will be "pip wheel; pip install
<the generated wheel>", bypassing setup.py install totally).

Paul