[Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))

[Vinay Sajip]

On Tue, 4/2/14, Carl Meyer <carl at oddbird.net> wrote:

> Sadly I can't offer that specific problem, because AFAIK we never
> tracked it down. It was a recurring problem in IRC where people
> would report using -E and pip would fail to take its action within
> the indicated virtualenv, instead impacting their global environment.
> After trying and failing a number of times to isolate and reproduce
> that problem, I decided not to continue spending time on that,
> and that it would be more robust to rely on the natural isolation
> provided by just using pip inside the env, which required no
> additional code to debug and maintain. (Pip had already been
> installed inside every virtualenv for quite a while by then.)
 
> I regret the way I handled that particular backwards-incompatibility
> (not providing a deprecation path), but I think the technical choice
> was a good one. Identifying code that is a bug-report-magnet and
> choosing a viable approach where that code can be removed
> entirely is a sound technical choice, not a matter of "politics" or
> "religion" (I had no political or religious motivations in removing
> pip -E).

I haven't made any comment about pip -E and how / why it was
removed - until Donald's post about it, I didn't even know about it.

To me, the correct solution would have been to isolate the bug and
remove it, but I completely understand the pragmatic approach you
took. But that leaves a "dark corner" where people might be afraid
to experiment in this area with pip, because it's somehow "scary".
This illustrates in part why working *on* pip is painful for me - it's
not exactly a large code-base, but the way it and its tests are
structured makes debugging it needlessly hard. In an environment
where this is all time-constrained volunteer activity, it is not at all
surprising that you took the action you did.

> As far as I can see, you haven't presented any technical issues with
> the current approach, just a feeling that it's inelegant (is this perhaps
> a religious feeling? :P). My feeling differs; I find it elegant to rely
> only on the virtualenv's built-in isolation, and inelegant to require
> extra restart-in-env code.

I didn't say that the current approach didn't work - it's just that IMO
it's inelegant to require something to be installed into every
virtual environment, just to be able to install packages into it. Seemingly
we've gotten so used to it, like "python setup.py install", that it seems
like part of the natural order of the universe :-)

It's not especially elegant to have to restart in a venv, but IMO it's the
lesser of two "evils". It is something that we currently have to do
(virtualenv does it too, with -p) but if a more elegant approach were
found, I'd certainly use it. The other inelegance (install pip in every venv)
is not essential, as I've shown.

I don't know exactly what approach pip took to implement -E - was it
similar to the approach in my snippet? Certainly, if anyone were to
point out any drawbacks with the logic in that snippet, I'd be interested.
But if pip took a different approach to implement -E, it's possible that
there was a problem with the details of that particular approach, and
the problems with pip -E don't provide any evidence to invalidate the
restart approach in general.

I agree that tastes differ, and technical arguments can perfectly
validly be about aesthetics in my view, but I don't regard these sorts
of arguments as "religious". If "religion" is to be mentioned in connection
with pip, then perhaps it's fair to say that pip is the "sacred cow" of Python
packaging ;-)

Regards,

Vinay Sajip