[Distutils] PEP470, backward compat is a ...

[holger krekel]

On Fri, May 16, 2014 at 13:38 -0500, Carl Meyer wrote:
> On 05/16/2014 12:10 PM, Donald Stufft wrote:
> >> 2. Add a deprecation path for --allow-unverified; can describe it in
> >> general terms as "the PEP 438 installer flag allowing installation of
> >> unverified external packages" if you don't want to be pip-specific.
> >> Currently PEP 470 has no mention of this, but I think letting a
> >> deprecation of --allow-unverified fully run its course _before_ making
> >> breaking changes on the PyPI side is a critical part of making this
> >> transition in a user-friendlier way.
> > 
> > Perhaps I wasn't being as obvious as I thought! My goal was to nail down what
> > the final destination looked like, and then once we figured out an end goal
> > deprecate everything that doesn't match that end goal (and ultimately remove
> > it).
> > 
> > One thing I don't want to do is add another intermediary fix with it's own
> > combination of flags that will do the thing that the user wants to do. We
> > already have:
> > 
> > 1) No additional flags
> > 2) --allow-external + --allow-insecure
> > 3) --allow-external + --allow-unverified
> > 4) --allow-unverified
> > 
> > Depending on what version of pip you're using[1]. I really really want the 5th
> > incantation of this to be the final incantation. That was one of the reasons
> > why I went the way I did in PEP 470. I don't believe it's possible to move
> > away from these without a break in compatibility for <=7% of projects,
> > *however* a key benefit of PEP 470 is that the mechanisms for allowing
> > additional locations has existed in pip for a long time. We can have a singular
> > clear message that says "If you want to do X then use these flags" and it
> > doesn't matter what version you're on. I vastly prefer that to the current
> > situation (and the "just let the deprecation run it's course" proposal) where
> > you have to pick the right combination of flags based on pip version.
> 
> I guess the key thing I don't understand yet is, why would we assume
> that any package that hasn't already switched to verified-external-links
> since PEP 438, given a one-year window so far to do so, is any more
> likely to populate this new discoverable-index metadata from PEP 470,
> given a six-month window?

That's a crucial question indeed (and i also agree to your earlier summary
of the situation, btw).

> It seems like PEP 470 places a lot of weight on an assumption of active
> maintainers, when really the core problem is a significant chunk of
> packages that (from the evidence of PEP 438 uptake) don't have active
> maintainers.
> 
> So if we conclude that the bulk of the problematic legacy packages will
> probably never populate this new discoverable-index metadata either, at
> what stage in the process would their users get any useful clue as to
> how to continue to install that package?
>
> One option is Holger's solution: scraping the current links and
> populating them as verified external links. We both don't like this
> because it involves PyPI misleading users about the status of those
> links, and you also don't like it because you want to deprecate verified
> external links too.

Regarding the last bit, i indeed don't want to see the possibility removed
to register external links-with-checksum.  PEP438 just introduced it and
i think it's sufficient and efficient for allowing maintainers to host
files externally while making it easy for end users choose to accept it
along with a slight loss of reliability that "--allow-externals" might
entail.  Integrity guruantees of such external links and pypi-uploaded
files are strictly the same.  If i understand correctly, PEP470 aims to
discontinue this PEP438 mechanism and i strongly disagree with that.

> Another option is if the deprecation message for --allow-unverified also
> gives the user the exact --find-links URL(s) they need to install that
> same package/version (which I think is implementable in pip today
> without any changes in PyPI). The downside here is that it really
> doesn't improve the current UX for these legacy packages, it just
> replaces --allow-unverified with explicit --find-links: but at least the
> latter is more explicit and places the responsibility clearly on the
> external host, not PyPI.

It also places the responsibility to have project-specific options
on the end user.  I prefer this path of action to the current PEP470, however.

> Or, thirdly, Paul's proposal could solve this, if PyPI automatically
> generated an "external legacy index" for any packages that haven't
> generated their own external index URL by a certain date. Really in a
> way this is similar to Holger's proposal, except it uses
> external-indexes instead of verified-external-URLs, and is again a bit
> more explicit about what's going on (at the cost of requiring more
> adjustment from users).

I find Paul's idea most interesting.  What about this pypi-hosted
legacy index providing "external" links so that you would still need to
say "--allow-external"?  I am imagining something like:

    pip install --extra-index https://pypi-stale-unsafe.python.org/ --allow-external

This index could be a statically served nginx site because
nothing would be added to it dynamically by users.  It would
contain all of the "auto-crawled" packages with the snapshot
checksums we found at the time.  And the "--allow-external"
signals that the files are not hosted on pypi.  No maintainer would
need to setup anything for his project and end users could add
this one extra index if they somehow depend on those packages (and 
get the hint from newer pip versions and from the PyPI main page).

Going down this route would allow to get rid of "--allow-insecure",
"--allow-unverified" and we'd only have the PEP438 "pypi-explicit"
hosting mode. 

best,
holger

> Basically, I think some acknowledgment of this problem of packages
> without active maintainers (and ideally a proposed solution to it)
> should be in PEP 470.


> Carl
> 



> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig