[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[M.-A. Lemburg]

On 12.05.2014 15:58, Paul Moore wrote:
> On 12 May 2014 13:28, M.-A. Lemburg <mal at egenix.com> wrote:
>>> So, some questions:
>>>
>>> 1. Is MAL aware that egenix-mx-base is not verifiably externally
>>> hosted[1], and if so, what is he asking for? Automatic download with
>>> no need for opt-in of unverifiable external downloads? That seems
>>> pretty much in conflict with the whole intent of PEP 438.
>>
>> What we are implementing is a proposal that I brought up before
>> PEP 438 was put in place:
>>
>> Instead of linking directly to all packages, we put up a verifiable
>> link to an index page with verifiable links, with the net effect
>> being that tools can verify the whole chain.
> 
> Thanks for clarifying. My main concern is that people do actually
> understand the requirements for "safe" external hosting (I discovered
> that I certainly didn't - it's quite complex!). From what you say, you
> do understand the requirements but have chosen not to follow them at
> this time. That's fine, I'm not trying to debate the validity of your
> choice. But in the context of PEP 438, that means that users of the
> egenix downloads will have to opt into unverifiable downloads in order
> to install using pip. We're working towards improving the user
> experience for end users who need to install unverifiable downloads -
> I'd expect that one consequence of this will be that we'll be better
> able to communicate the situation to those users, which will reduce
> the confusion.

If it helps convince you that allowing verifiable external
links per default is a good thing for user experience, we will
register the distribution file download URLs with the PyPI
web API.

> I don't know if you're suggesting that your proposal is still
> something that we should be looking at implementing. I'm happy to
> discuss that, but it should probably be a separate thread.

You can think of it as per-package PyPI-style simple index
that's registered with PyPI in a secure way (via the check sum
hash).

There's most certainly room for improvement and I'm open for
discussions.

>> since shifted focus to working on a web installer which solves
>> multiple problems at once:
> [...]
>> With the web installer, we'd just have to upload one file
>> per release.
> 
> I'm not quite sure how you expect this will work, but it's probably
> important that you get involved with the various packaging PEPs. The
> only way I can see such a solution working with pip would be if you
> have a customised setup.py.

Yep, since that's the way installers (not only pip) work when
they see a source distribution.

> As the general trend is towards binary
> installs using wheels, with pip eventually deprecating sdist installs
> and running setup.py directly, we will likely miss your use case
> unless you get involved in those discussions (they are on the back
> burner at the moment, but will likely resurface at some point -
> there's currently a work-in-progress PR for pip to use a "wheel
> cache", where the normal install route will be "pip wheel; pip install
> <the generated wheel>", bypassing setup.py install totally).

Binary installs are nice, but they are not the answer to everything
and no matter how much meta data you put into static files,
there will always be cases where that meta data description just
doesn't include those bits you would need to complete the setup,
esp. for packages with C extensions and more complex external
dependencies or setup requirements. (*)

The setup.py interface makes all this possible, which is why so
many Python packages use it to configure themselves automatically.

Deprecating this interface would make some distributions impossible
to install without manual user intervention and we'd be back to the
Makefile.pre.in days.

I don't think that's a good idea. It still is a very good idea
to make more meta data available in static non-executable form
in order to simplify finding packages, determining
dependencies, features, enhancing the PyPI UI, and for
deciding which distribution file to download and install.

This can be generated by setup.py as part of the build process
and made available to PyPI during package release registration
(much like it is now, only in extended form).

(*) This does work if you are only targeting a few select systems and
system versions, but the Python user base out just has too many
diverse setups to be able to address them all to be able to
completely drop setup.py.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/