[Distutils] The future of invoking pip

[Nathaniel Smith]

On Thu, Nov 5, 2015 at 1:08 PM, Donald Stufft <donald at stufft.io> wrote:
> Another possible option is to modify pip so that instead of installing into
> site-packages we instead create an "executable zip file" which is a simple zip
> file that has all of pip inside of it and a top level __main__.py. Python can
> directly execute this as if it were installed. We would no longer have any
> command except for ``pip`` (which is really this executable zip file). This
> script would default to installing into ``python``, and you can direct it to
> install into a different python using a (new) flag of ``-p`` or ``--python``
> where you can specify either a full path or a command that we'll search $PATH
> for.

I'm not sure if I like this idea or not, but I think it's an
interesting one. I'd frame it differently, though. It seems to me that
the core idea is:

A basic problem that pip has to solve, before it can do anything else,
is that it has to identify the python environment that it's supposed
to be working on. Right now, the standard way it does this is by
looking at sys.executable -- so pip is always doing an odd dance where
it's rearranging the universe around itself, and conversely, every
environment needs to have its own copy of pip inside itself. An
alternative approach would be to totally decouple the host python used
to execute pip from the target python that pip acts upon, on the
grounds that these are logically distinct things. (As a thought
experiment you can even imagine a package manager called, say, 'cpip',
which is a standalone program written in C or some other
non-Python-language, but that happens to know how to manipulate Python
environments. I'm not saying porting pip to another language makes is
in any way a good idea, just that imagining such a thing is a useful
exercise to clarify the difference between the host environment and
the target environment.) In this approach, you'd have a program called
'pip', and when run it uses some set of rules to figure out which
environment it should work on, consulting command line arguments,
environment variables, the current PATH, whatever.

(Windows has its own complicated system that I don't understand, but
for Unix-likes, the default rule would probably be: search the path
for "python" + sys.argv[0][3:], and use that. So you could install a
main pip executable + symlinks named pip3, pip3.4, etc., and invoking
each of them would automatically target the same environments you get
when you run python, python3, python3.4, etc.)

Having decoupled things like this, then it doesn't really matter how
pip is distributed, so long as there is a pip program and it works.
It'd be in a similar position to, say, how mercurial is shipped. You
could use pyinstaller to make a fully standalone package for Windows
that didn't even depend on the system python install, and Debian could
ship a version that's hosted by the system default python, and if you
want to have multiple pip's installed then you could do that just like
you do now by installing them into their own venvs.

Other interesting consequences:

- pip could get more aggressive about dropping *host* support for
older versions of python -- e.g. eventually pip itself could be
written in pure python 3 while still preserving support for installing
stuff into a python 2 target environment. Or less aggressively, you
could imagine dropping host support for 2.6 ahead of dropping target
support for 2.6.

- If pip is a thing that lives "outside" environments instead of
"inside" them, then it might be natural for it to grow some interface
tools for working with environments directly, becoming kinda the
one-stop-friendly-UI for Python developers. E.g.

   pip new-env my-env/ -p pypy3 -r requirements.txt

to create a new virtual environment and install stuff into it in one
go. (Probably pip is going to have to gain some related functionality
anyway to install setup-requires into isolated build environments...)
Obviously there's a desire not to shove everything in the world into
pip, but having a single friendly frontend to
installation/virtualenv/venv is *really* nice for newbies. And
obviously this would a be "someday maybe" thing at best, just an
interesting possibility down the road.

- The eventual interface seems nice enough...

path/to/python -m pip
-> pip -p path/to/python

pip3, pip3.5
-> still would work in my suggested interface

path/to/venv/bin/pip
-> pip -p path/to/venv/bin/python
or pip -E path/to/venv

though the transition would be tricky/painful.

I don't have any conclusion, I just think it's just an interesting
idea to think about.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org