[Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Paul Moore]

On 21 October 2017 at 12:15, Nick Coghlan <ncoghlan at gmail.com> wrote:
> (Note: this is entirely speculative, and I have no idea how hard it would
> be, so please read it as the question it's intended to be)

No problem - I don't know myself how hard some of this would be, either ;-)

> Do you know if there any key APIs (like installation) that could be turned
> into wrappers around pip CLI calls in order to mitigate some of the impact?

The obvious one is pip.main(). That's the one a lot of people use, but
it's easily replaceable by a simple subprocess call. That's actually
one of the reasons this was so frustrating to us - the bug reports we
got were often from people doing things they didn't need to, that they
could handle trivially via a supported approach.

Otherwise, no. We've had little or no feedback on the tracker from
people using more complex internals, so our working assumption has
been there's very little that can't be handled via the CLI or existing
packages. Feedback so far from this mail hints that maybe we were
wrong, but it's still hard to know if it's one or two key projects, or
a whole range of people that we've yet to hear from. I'm pretty sure,
for example, that pipenv uses internals, either directly or via one of
their dependencies, but we've not seen any feedback from them yet.

> The reason I ask is because it's unlikely anyone else is going to understand
> how to emulate the previous functionality better than the pip devs would,
> and if there's an API for those particular invocations, than they can be
> covered directly by pip's test suite.

Understood. We understand *how*, but don't know what is needed. One of
the points of all this is to tease out such requirements. We'd hope to
get them addressed by including them into *other* packages like
distlib or packaging, but that's mostly just a matter of where the API
goes, not what is needed. It does help us avoid tying fixes to pip's
release cycle, though.

It's also worth noting that the pip devs are possibly way too deep
into how pip does things, rather than what the standards say. Getting
others to implement libraries based on the published standards would
help immensely to ensure that we're not falling into the
"implementation defined" trap of the community being stuck with having
to use pip because no-one else knows how to do what pip does.

> Plus if there are previous API capabilities that *can't* currently be
> emulated via the CLI, then the pip devs are the folks in the best position
> to add the necessary CLI enhancements (such as the ones Noah asked about for
> doing a more selective `pip list`).

Oh, sure - apart from the aforementioned "pip.main()", most
capabilities of the internal API are *not* easily emulated by the CLI.

But 3rd party libraries are just as much an option. Remember, the
issue here isn't so much about designing an API as about exposing (and
therefore locking in stone) internal implementation details of how pip
does things. And the pip devs are arguably in the *worst* position to
handle that option, precisely because they know so much about how pip
does things.

> If that's an approach you might be amenable to, then a 10.0 pre-release
> could be a good time to solicit PRs from folks that were using particular
> APIs and would be prepared to invest time in defining comparable CLI
> wrappers for them.

Well, I get your point here, but the implication of this is that we
have to design and build an API before we can release pip 10. Calling
it CLI wrappers (and implementing it that way) doesn't do anything to
ease the work needed on design, or the maintenance burden of providing
stability. We're already getting a lot of pressure to release pip 10,
and trying to do that would push it way further off.

To be blunt, no-one on the pip team is really interested in trying to
provide or support a stable API at this point in time. We have our
hands full, as everyone is aware, and this is way down all our
priorities. Community-submitted PRs would help, but there's still work
in agreeing a design, reviewing those PRs, and maintaining them
(exposing details from the internal APIs limits how much we can change
those internals later, which is something we have to consider).

What would be ideal would be for the community to build
standards-based libraries that satisfied the needs that people
currently handle by using pip internals. Heck, we could even consider
vendoring such libraries inside of pip and saving ourselves a bunch of
complexity ;-) The packaging, pkg_resources and distlib libraries are
examples of this happening. But it doesn't seem to be a popular route,
unfortunately. And those particular libraries are all maintained by
PyPA members, so don't really count as examples of the community
getting involved...

The most likely alternative solution would be to revert the internal
API moves. If that happened, I'd still prefer to lose pip.main, as
that's the major pain point for us and the easiest for users to solve,
but we could put the rest back. But it would be naive to assume that
as a result people like Noah would be unaffected - they'd actually be
worse off, as the internal APIs they use would probably break anyway
(the installer/resolver stuff has been heavily modified for pip 10)
but not all would do so in obvious ways.

Beyond this, I don't know.

Paul