[Distutils] DWIM installation with setuptools

[Jim Fulton]

Phillip J. Eby wrote:
> Okay, so I've been thinking about all the recent complaints/requests 
> regarding the perils of PYTHONPATH, site-packages, --prefix, and all 
> that. 

Yay! :)

> And I'm thinking, is there some way I can work around all this stuff
> so that installation is totally DWIMmish   -- that is, that the install can
> just "do what I mean", while still allowing a default version of a package
> to be set.

Hm.  I don't like this goal as stated.  I'm not looking for total DWIM.  I'm
looking for an efficient way to control what's going on.

For example, when setting up an application directory, I don't mind
running some utility with explicit options when I set it up
or creating a configuration file with explicit options in it.
(I am a bit annoyed by use of the current working directory to
find setup.cfg, but I can live with it if it only matters when
using easy_install).

> 
> This already works perfectly with root access or a virtual Python, so we 
> can pretty much ignore those setups for this discussion (except to make 
> sure we don't break them).
> 
> So, mostly what's left to worry about is being able to select an arbitrary 
> installation location, and still have it work. 

Yup, where the selection is through some explicit means.

> If we assume you're smart
> enough to pick a location that will already be on your PYTHONPATH (and we 
> could verify that by checking that it already is), then all we have to 
> worry about is making sure that the right version of a package gets added 
> to sys.path at runtime.

I don't like this assumption.  I don't want to make people set PTHONPATH
to make this work, especially for generated scripts.  I'd rather have
the scripts generated to do necessary manipoulations themselves.
That is, I don't mind setting the path, if necessary, when I run easy_install,
but I don't want users to have to set the path when running the scripts.

> For scripts this isn't that big of a deal.  I could create a specialized 
> bootstrap loader version of pkg_resources.py and dump it alongside any 
> scripts installed by easy_install.  It'd be a pain to implement, but I 
> could probably make it work such that sys.path was searched for a 
> setuptools egg, and then have that get used to bootstrap the rest.

Sounds promising.

> What this doesn't work for is the interactive interpreter,

Good point.  Although, perhaps irrationally, I can live with modest path
setting when running the interactive interpreter.

> or code that
> doesn't use require(). 

Isn't that a separate issue?  Are you refering to non-generated scripts?

> Even if the relevant directory is on sys.path,
> Python doesn't normally read .pth files in PYTHONPATH directories.  I've 
> also learned that many Linux distros appear to hack sys.path in odd ways, 
> and I'm not positive that all of them process .pth files in the directories 
> they add.  Certainly they don't support .pth for PYTHONPATH directories.
> 
> Since it's not just the interactive interpreter, hacking .pythonrc isn't 
> going to help much. 

But if we come up with a solution that handles everything but the
interpreter, hacking .pythonrc might be a nice way to fill the gap.

> That leaves the 'site' and 'sitecustomize'
> modules.  The current PYTHONPATH install support hijacks the 'site' module 
> with a special .pth-honoring feature, so as long as you have the setuptools 
> egg listed directly in PYTHONPATH, you're golden.

Yup.  Of course, anytime you make a copy of something, you introduce a
problem.  This is a big part of my objection to the virtual python approach.

> But alas, that isn't DWIM, since you now have to manipulate the PYTHONPATH 
> *before* you can reasonably do anything.  Part of the essence of what we 
> desire is that any required steps be done for you *during* the installation 
> rather than requiring an extra step before or afterwards.

Yup.

> Unfortunately, this means that hacking 'site' can't work unless we actually 
> install a site.py.  Will that work?
> 
> Well, yes and no.  It appears Python will recognize an added site.py that 
> appears in a PYTHONPATH, but many platforms will not recognize a site.py 
> that's in the script directory but not PYTHONPATH.  Windows seems to be the 
> only platform where putting a site.py in the script directory has any effect.
> 
> This is really good news, however.  It appears to mean that slapping a copy 
> of my hacked 'site.py' (i.e., the one that honors .pth files along 
> PYTHONPATH) into the easy_install target directory would make 
> PYTHONPATH-based installs essentially DWIMmish.  To complete the illusion, 
> I'd need to add some code to script wrappers to process the setuptools.pth 
> file in the script directory if pkg_resources can't be otherwise imported.
>
> We could then lobby for the Python 2.5 site.py to behave the same way as my 
> hacked one does: i.e., process .pth files found in PYTHONPATH directories, 
> inserting them ahead of site-packages and its ilk, so that it wouldn't be 
> necessary to install the hacked site.py in such cases.  (It's also not
> necessary to install it in known "site" directories, such as site-packages 
> and any directories listed in --site-dirs.  But even if it's installed 
> unnecessarily, it shouldn't hurt anything.)
>
> Since 'site' is a stdlib module and isn't normally intended for 
> customization, it shouldn't conflict with any user-installed 
> modules.  However, easy_install could detect an existing site.py and warn 
> if its content is different from the one easy_install wants to use.
> 
> I'm trying to remember why I didn't do this around the time I implemented 
> the original PYTHONPATH support (which first required a hacked 'site' 
> module).  I think I was thrown by the lack of cross-platform script 
> directory support for importing 'site', and it didn't occur to me that I 
> could easily work around that from within scripts.  Also, it's easy to 
> mentally confuse the script directory and the install directory while 
> thinking about some of this stuff.
> 
> But as long as the install directory (irrespective of the script directory) 
> is on PYTHONPATH and has a hacked site.py, what's in the script directory 
> doesn't matter.  The only case where the script directory matters is if:
> 
> 1. it's the same directory the libraries were installed to,
> 2. it's not on PYTHONPATH, and
> 3. the user puts a script of their own in there and expects it to work 
> anywhere but Windows
> 
> This edge case could be handled, however, by refusing to install libraries 
> to non-PYTHONPATH directories unless you're on Windows or you use 
> --multi-version.  I'm somewhat uncertain about this, though, because it 
> doesn't seem to let you create a self-sufficient application directory.
> 
> Right now, you can use "easy_install -ad somedir ..." to copy all needed 
> eggs and scripts to the target directory, making a self-contained 
> application directory that has everything it needs to run.  OTOH, I suppose 
> if you just made that -mad instead of -ad, it would work just fine for that 
> purpose under the proposed new regime.
> 
> So, to sum up, here are the changes that would take place:
> 
> * easy_install would treat all PYTHONPATH-specified directories as though 
> they were listed in --site-dirs, but it would also copy and compile a 
> hacked site.py into them during installation, if they aren't *actually* 
> listed in --site-dirs or in setuptools' default computation of "site" 
> directories.
> 
> * Currently, easy_install defaults to -m if you install to a non-"site" 
> directory; it would now instead stop and refuse to install to such 
> directories unless you explicitly specify -m.
> 
> * easy_install could now meaningfully grow a --prefix option, since it will 
> now DWIM as long as the target dir is recognizable as (or convertible to) a 
> "site" directory.  And, in the cases where DWIM can't be guaranteed, it 
> will stop and inform the user that they need to fix PYTHONPATH, 
> --site-dirs, or use -m/--multi-version mode.
> 
> What do y'all think?  Do we have a winner yet?

I think you're on the right track, at least from a functionality point of view,
although I don't want to require a special PYTHONPATH setting to
use a generated wrapper script.

I do think we should be a bit more conservative though.  I was already a bit
worried about the current use of a copied/hacked site.py. I don't think it's a
good idea to take it further.  That kind of hackery tends to turn people off
with good reason.

Maybe we should take a step back.  Reading about all of these hacks
convinces me we don't understand the problem well enough or that we
need more experience trying different solutions.  I fear that some of
the proposed solutions are too complicated.

Increasingly, I think maybe all I want/need is a hook, perhaps one that
I already have. :)  Two possibilities:

- I could look at alternate script generation strategies myself.
   I suppose I could write my own easy install wrapper myself that
   tried to do that.  It would probably be a good learning exercise
   for me.  That is, someone would use my wrapper script to install an
   egg.  My script would call easy_install, telling it not to generate
   scripts.  It would then instrospect the installed egg and it's
   dependencies and install scripts itself.  Is there some sort of hook
   I can use to introduce custom logic into the install process?

- It occurs to me, based on some of your suggestions, that it would be enough
   for my needs if generated scripts always looked for some Python code
   in the same directory as the script and, if found, executed it.  This need
   not be a module.  It might even be better if it wasn't, but I'm not sure.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org