[Python-Dev] Proposal for virtualenv functionality in Python

[Ian Bicking]

On Sun, Feb 21, 2010 at 12:32 PM, Larry Hastings <larry at hastings.org> wrote:

>  * I'd rather ~/env/bin/python be a symlink instead of copying it.
>>
>
> The thread discussing Windows suggests that we shouldn't use symlinks
> there.  I'd say either copying or symlinking pythonv should be supported,
> and on Windows we recommend copying pythonv.exe.


Sure, on Windows this is clearly the case.  I'm not sure if it's worth
supporting elsewhere.  One problem with copying is that (a) you don't know
where it is copied from (requiring extra information somewhere) and (b) if
there is a minor Python release then things break (because you've copied an
old interpreter).  Probably there should be a check to catch (b) and print
an appropriate (helpful) error.


>  * Compiling extensions can be tricky because code may not find headers
>> (because they are installed in /usr, not ~/env/).  I think this can be
>> handled better if virtualenv is slightly less intrusive, or distutils is
>> patched, or generally tools are more aware of this layout.
>>
>
> Conversely, headers may be installed in ~/env and not /usr.  The compiler
> should probably look in both places.  But IIUC telling the compiler how to
> do that is only vaguely standardized--Microsoft's CL.EXE doesn't seem to
> support any environment variable containing an include /path/.
>
> I suspect solving this in a general way is out-of-band for pythonv, but I'm
> willing to have my mind changed.  Certainly pythonv should add its prefix
> directory to LD_LIBRARY_PATH on Linux.


Yes, it might be possible to change distutils to be aware of this, and some
things will work okay as a result.  Some things will really require changes
to the problematic project's setup.py to support this.



>
>  Additionally, the binary will look for a configuration file.  I'm not sure
>> where this file should go; perhaps directly alongside the binary, or in some
>> location based on sys.prefix.
>>
>> The configuration file would be a simple set of assignments; some I might
>> imagine:
>>
>> * Maybe override sys.prefix
>> * Control if the global site-packages is placed on sys.path
>> * On some operating systems there are other locations for packages
>> installed with the system packager; probably these should be possible to
>> enable or disable
>> * Maybe control installations or point to a file like distutils.cfg
>>
>
> I'm unexcited by this; I think simpler is better.  pythonv should
> virtualize environments layered on top of python, and should have one
> obvious predictable behavior.  Certainly if it supports a configuration file
> pythonv should run without it and pick sensible defaults.
>
> What are the use cases where you need these things to be configurable?
>

* Override sys.prefix: allow you to put the binary in someplace other than,
say, ~/env/bin/python and still support an environment in ~/env/.  Also the
use case of looking for libraries in a location based on the interpreter
name (not the containing directory), like supporting /usr/bin/python2.7 and
/usr/bin/python2.7-dbg.
* Control global site-packages: people use this all the time with
virtualenv.
* Other locations: well, since Ubuntu/Debian are using dist-packages and
whatnot, to get *full* isolation you might want to avoid this.  This is
really handy when testing setup instructions.
* Control installations: right now distutils only really looks in
/usr/lib/pythonX.Y/distutils/distutils.cfg for settings.  virtualenv
monkeypatches distutils to look in
<sys.prefix>/lib/pythonX.Y/distutils/distutils.cfg in addition, and several
people use this feature to control virtualenv-local installation.


>
> Let me propose further about python and pythonv:
>
>   * As Antoine suggested, the CPython interpreter should sprout a new
>     command-line switch, "--prefix", which adds a new prefix directory.
>

OK; or at least, it seems fine that this would be equivalent.


>   * pythonv's purpose in life is to infer your prefix directory and
>     run "pythonX.X --prefix <prefixdir> [ all args it got ... ]".
>

I don't see any reason to call the other Python binary, it might as well
just act like it was changed.  sys.executable *must* point to the originally
called interpreter anyway.


>   * Should pythonv should be tied to the specific Python executable?     If
> you run install pythonv as "python", should it look for
>     "python" or explicitly look for the specific Python it shipped
>     with, like "python3.2"?  I suspect the latter though I'm no longer
>     sure.
>

Experience shows the latter, plus this would only really make sense if you
really called the other interpreter (which I guess you could if you also
added an --executable option or something to fix sys.executable).  If you
did that, then maybe it would be possible to do with PEP 3147 (
http://www.python.org/dev/peps/pep-3147/) since that makes it more feasible
to support multiple Python versions with a single set of installed
libraries.  3147 is important (and that it be backported to 2.7).

I have to think about it a bit... but maybe with this it would be possible
to move these environments around without breaking things.  That would be
compelling.


> I'm one of those folks who'd like to see this be stackable.  If we tweak
> the semantics just a bit I think it works:
>
>   * pythonv should inspect its --prefix arguments, as well as passing
>     them on to the child python process it runs.
>

With a config file I'd just expect a list of prefixes being allowed;
directly nesting feels unnecessarily awkward.  You could use a : (or
Windows-semicolon) list just like with PYTHONPATH.


>   * When pythonv wants to run the next python process in line, it
>     scans the path looking for the pythonX.X interpreter but /ignores/
>     all the interpreters that are in in a --prefix bin directory it's
>     already seen.
>   * python handles multiple --prefix options, and later ones take
>     precedence over earlier ones.
>   * What should sys.interpreter be?  Explicit is better than implicit:
>     the first pythonv to run also adds a --interpreter <argv[0]> to
>     the front of the command-line.  Or they could all add it and
>     python only uses the last one.  This is one area where "python" vs
>     "python3.2" makes things a little complicated.
>
>
Ah, yes, the same problem I note above.  It should definitely be the thing
the person actually typed, or what is in the #! line.


>
> I'm at PyCon and would be interested in debating / sprinting on this if
> there's interest.
>

Yeah, if you see me around, please catch me!


-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://twitter.com/ianbicking
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100221/307f8c23/attachment-0001.html>