[Distutils] some questions about PEP470

[Donald Stufft]

> On Oct 13, 2014, at 7:41 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote:
>> On 12 October 2014 09:49, Donald Stufft <donald at stufft.io> wrote:
>>> 
>>> On Oct 11, 2014, at 7:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> 
>>>> On 12 October 2014 04:29, Donald Stufft <donald at stufft.io> wrote:
>>>>>> I plan to put the external repositories (and the commands needed to use them)
>>>>>> in the UI for PyPI. I suppose I should put that in the PEP as well, I was more
>>>>>> focused on defining the API differences and the changes.
>>>>> 
>>>>> I forgot to mention, one other thing I thought about, we could move the current
>>>>> external links to external repositories automatically. I’m not sure if this is
>>>>> a good idea because on some projects that will end up with a giant list of things
>>>>> that a user would be suggested to install. Perhaps it’d work to scan them for
>>>>> installable files and only move the ones where an installable file was discovered,
>>>>> though that could be error prone (for example right now bytereef.org is down, so
>>>>> we’d not discover any files there).
>>>> 
>>>> If I recall Holger's suggestion correctly, it was just to move the
>>>> current "Download URL" links over, rather than the scraped links.
>>> 
>>> Hmm, I thought his suggestion was to just leave the existing links alone so that the
>>> existing pip/easy_installs would continue to function as they do now?
>> 
>> Sure, we could do that too. Regardless, I don't believe we should let
>> the conversion of the silent security failures into noisy exceptions
>> block the other usability enhancements - if decoupling the two
>> decisions will achieve that, then let's go down that path.
>> 
>> I think I suggested at one point that the PEP could potentially commit
>> to making the backwards compatible changes on the server side, and
>> then for the *in*compatible changes, only commit to *looking at the
>> numbers again* in a few months time after folks have had a chance to
>> experience the replacement model. Client applications would be
>> officially granted permission to ignore the PEP 438 external hosting
>> mechanism in favour of the new PEP 470 external index advertisements.
>> 
>> The sequence of events from a compatibility break/user experience
>> enhancement perspective would then be:
>> 
>> * new model added to PyPI (no compatibility break)
>> * old model dropped from pip (client side compatibility break, error
>> handling improvements)
>> * old model dropped from PyPI (conditional, based on changes in server metrics)
> 
> I like this strategy and would add that I guess it's then only a matter of
> when not if the old model is dropped (and the PEP could encode that).

This is more or less the sequence of events that will happen with PEP 470
as it is except that PEP 470 doesn’t make the deprecation optional. I do
not believe it is a good idea to make it optional because it is a silent
security risk. The other main difference is that PEP 470 doesn’t allow new
projects to ever register externally hosted files in the old way.

> 
> The question is how to best do the "(almost) no compat break" for
> a reasonable definition of "almost".  I am mostly concerned about
> the verified externals and "crawl urls".  If they keep showing up as
> "rel=download" links on the simple page that should be fine for existing
> tools, right?

Keeping backwards compatibility for unsafe things is completely off the
table as far as I’m concerned. Specifically that means the crawl links.
I think keeping backwards compatibility for the ~70 projects which are
doing things safely today is a waste of resources in the long run, but
something that should be phased in as PEP 470 does. More people were
broken when PyPI went forced TLS and then again when PyPI switched from
a sha1 certificate to a sha256 certificate. The numbers of safe installs
is miniscule.

> 
> But with PEP470 they might get an additional attribute so that a newer
> pip can recognize it and advise accordingly.  How pypi/warehouse
> internally keeps the state during/post PEP470 is less clear to me but it
> might involve a one-time conversion from the existing information to the
> new one.
> 
> I am actually wondering if a new pip could change its UI even without
> any changes to pypi.python.org? For example, when "pip install pil"
> happens it would say:
> 
>     Could not find any release file for "Pil" on https://pypi.python.org 
>     itself but the project has historically had additional external pages 
>     which linked to release files:
> 
>        http://effbot.org/zon/pil-changes-115.html/
>        http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz
>        http://www.pythonware.com/products/pil
> 
>    Please manully check these links if they directly link to or contain
>    release files for "Pil" and then add an appropriate one like this::
> 
>        pip install --find-links URL [other options] Pil
> 
>    (See <link> for more info ...)
> 
> and pip could deduce this information from the current https://pypi.python.org/simple/pil/page.  (The pip message above may not be precisely right but
> you get the idea).
> 
> FWIW this strategy of advising to use newer/other options but keeping
> the old working (undocumneted or documented with a warning)
> is how i generally try to handle things with pytest.  It allows
> people to use the new way and incrementally migrate their setups.
> And if someone complains about problems with the old ways, point them to
> the new ways so there is no need to maintain the old ways too much.

It’s a fine way of doing thing, and we do it in pip too, when (and this is
important), it’s not a security sensitive issue.

> 
> Even then, it would probably still makes sense to advance the server
> side and allow explicit registration of external index pages i guess
> (and maybe findlinks pages so you can really just use apache/nginx
> directly?) and then have pip provide a more precise message.
> 
>> That changes the dynamic from "the updated user experience will be
>> better, trust us" to "if you prefer the old user experience, you can
>> keep using the old pip, if you prefer the new user experience,
>> upgrade".
> 
> I think it's very much fine to say "this is the future" as long as
> old-tool/current-server instances keep working for a longer while.
> 
> Python packaging infra is a bit like a big container ship these days --
> which is both a good and a bad thing i guess -- and people build
> all kinds of funny containers on top.
> 
> best,
> holger
> 
>> The eventual decision on when to drop the legacy model from PyPI can
>> then by made based on a richer set of input data, that includes the
>> level of adoption of the new version of pip that treats all missing
>> link targets as potential errors.
>> 
>> Regards,
>> Nick.
>> 
>> -- 
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA