[Distutils] pythonv, take two

[Carl Meyer]

On 03/30/2011 07:18 PM, Vinay Sajip wrote:
>> So... it seems to me that we're likely to break _some_ third-party code
>> using sys.prefix regardless of what we do here. My instinct says adding
>> sys.virtual_prefix and leaving sys.prefix alone is the better approach,
>> but I'm not very firmly entrenched in that position.
> 
> That's problematic, of course (third-party breakage, I mean). It may be that
> despite using virtual_prefix being a cleaner approach, it may be necessary
> to alter sys.prefix to ensure compatibility, a la virtualenv.

Well, as I said, I think there will be compatibility problems either
way. We are splitting site-packages from the base Python installation,
and when third-party code assumes that site-packages and the rest of the
base Python installation are both found relative to sys.prefix, neither
choice we make can possibly be right.

Virtualenv tries to avoid this problem by copying or symlinking in
enough stuff from the base Python install in order to convince third
parties using sys.prefix that they really are looking at a full Python
installation. But that approach is hacky and far from problem-free, and
requires keeping up with various things that third-party code might try
to look for under sys.prefix (there is at least one current open bug on
virtualenv, Tkinter failing in a Windows virtualenv, that is the result
of this problem). The logical conclusion of that approach, if you want
full compatibility with third-party code, is to just copy the entire
Python installation to the new location.

So three options as I see it:

1) Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better
name), and require third-party code to use that (or sysconfig) to find
site-packages, if it wants to be virtual-compatible.

2) Repoint sys.prefix, and require third-party code to understand some
new sys.system_prefix or sys.installed_prefix or equivalent to find the
rest of the base Python installation.

3) Attempt the virtualenv approach of trying to fool third-party code
into thinking the virtualenv IS a full Python installation, even though
it's not.

I think #1 is far preferable to #2, and in the long run preferable to #3
even if it involves some short-term pain. But maybe I'm being too
idealistic, and #3 is the pragmatic choice even though it's "wrong."

> It may be that the thing isn't working properly, but it's masked on my
> system because I have an installed 3.3 in /usr/local (from when I was doing
> my pip/virtualenv tests). This means that there is a Makefile in a location
> which would not be there if I'd never installed 3.3 using sudo make install.
> 
> However, the getpath.c stuff should work - to test this, insert an
> "import pdb; pdb.set_trace()" in site.main, and check sys.path. This should
> contain entries from the virtualenv. The problems are more likely to be in
> the Python code: the C code always sets sys.prefix and sys.exec_prefix to
> /usr/local (or whatever configure was invoked with) and the sys.prefix /
> sys.exec_prefix stuff is done in site.virtualize, as before.

No, if I break early in site.py (or run python -S), sys.path doesn't
contain any virtualenv paths.

> Sorry - I got in a muddle because I've got too many clones around and ran
> into free disk space issues, and goofed somewhere. Don't bother to do a full
> merge for now, as I'm not sure if my code's working yet. Since I don't expect
> to change more than 4 files (getpath.c, site.py, distutils/sysconfig.py and
> sysconfig.py) I don't think we'll have too much trouble merging in due
> course, and if/when we can agree that my experiment in pythonv is worth
> taking further, I'll do the work of merging into cpythonv. For now, though,
> my changes might be treading on your toes, making merging needlessly
> annoying.
> 
> I'll try to test on a Python-clean machine in a day or two, will report back
> then.

Ok, thanks!

Carl