[Distutils] pythonv, take two

[Carl Meyer]

On 03/31/2011 04:12 AM, Vinay Sajip wrote:
> Carl Meyer <carl <at> oddbird.net> writes:
> 
>> The logical conclusion of that approach, if you want
>> full compatibility with third-party code, is to just copy the entire
>> Python installation to the new location.
> 
> We do want that compatibility, don't we? Using a virtual environment should be a
> no-brainer, as virtualenv currently is (most of the time). Of course, the
> downside of the copying would be around 50MB per virtualenv on Windows, and much
> less where symlinking or hardlinking is available. Not ideal, of course, but
> disk space is cheap ;-)

People tend to create a lot of virtualenvs. I don't think copying the
entire Python installation is viable (it is slow, and it also would mean
that security updates to the base standard library would not be
reflected in the virtualenv). If copying the entire base Python install
were an adequate solution, this entire virtual-python proposal would be
moot - you can do that right now and it works fine.

>> 1) Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better
>> name), and require third-party code to use that (or sysconfig) to find
>> site-packages, if it wants to be virtual-compatible.
> 
> But the success of virtualenv is because you can install just about any package
> in an environment, and packages don't have to do anything to be "virtual
> compatible".

Sure. "virtual-compatible" was a poor choice of terms - in the long run,
there should be no such concept. In the short run, we would be
clarifying the semantics of sys.prefix and introducing a new
sys.site_prefix. And in the short run, the effect of code that used
sys.prefix contrary to those semantics would be failure in a virtualenv.

But, as I mentioned, the fact that easy_install itself relies on finding
site-packages relative to sys.prefix forces the issue - easy_install
(including existing versions of it) needs to just work.

>> 2) Repoint sys.prefix, and require third-party code to understand some
>> new sys.system_prefix or sys.installed_prefix or equivalent to find the
>> rest of the base Python installation.
> 
> I think that the answer is for third party code not to use any prefix directly,
> but rely on an API such as  sysconfig.get_path('purelib'). There's a distutils
> version of this - distutils.sysconfig.get_python_lib() - but this is brittle as
> per issue #6087.

Yes, of course. If all code did this, we wouldn't have this problem; we
would just fix sysconfig.py and site.py and all would be well.

>> 3) Attempt the virtualenv approach of trying to fool third-party code
>> into thinking the virtualenv IS a full Python installation, even though
>> it's not.
> 
>> I think #1 is far preferable to #2, and in the long run preferable to #3
>> even if it involves some short-term pain. But maybe I'm being too
>> idealistic, and #3 is the pragmatic choice even though it's "wrong."
> 
> I'm afraid you may be right, at least until people stop using sys.prefix and
> sys.exec_prefix directly and call functions instead, which can operate
> differently in a virtual environment.

Yup.

Carl