[Distutils] EasyInstall: bootstrapping

[Phillip J. Eby]

At 04:05 PM 6/4/2005 -0500, Ian Bicking wrote:
>Phillip J. Eby wrote:
>>Hm.  I would suggest that the safest tack to take is try importing 
>>pkg_resources and do a 'require()' for the setuptools version that you 
>>need.  If the import fails or the version fails, bail out of setup.py, 
>>asking for the user to install the correct setuptools.  I think that 
>>trying to invoke easy_install recursively or guess where the egg should 
>>be installed is a bad idea.  (E.g., guessing about the egg location will 
>>cause easy_install's sandboxing to shut you down anyway)
>>However, it would certainly be nice to have a way for setuptools-using 
>>packages to bootstrap the setuptools install -- and that includes 
>>setuptools itself.  (My head is starting to spin now.)
>>Hm.  Unfortunately, it seems that the only other sane way to do this is 
>>to distribute Paste as an egg, and require people to use easy_install to 
>>install it, because then any dependency back to setuptools can be 
>>resolved properly once easy_install's automatic dependency installation 
>>is in place.
>
>I wouldn't mind so much requiring easy_install to be installed -- if 
>that's the only firm prerequesite, that's okay.  But a Paste checkout 
>gives you docs and examples and unit tests, which aren't part of an egg, 
>so Paste as installed by easy_install isn't complete.

Well, there's always -b, as of 0.3a3, which keeps the unpacked source 
distribution after installation.  Does that help?


>One option would be if there was an option to install something the same 
>way you recommend using a development checkout -- don't create an egg, 
>just make sure the PackageName.egg-info directory is in place.

Ugh.  I don't want to do that for stuff that isn't either a development 
checkout or part of the standard library.  Anywhere in between and you 
start to mess up version management.  Note that when an .egg-info directory 
is in a directory on sys.path, it means the egg in question is *already* on 
sys.path, which means you can't require() a version that conflicts with it.

So, .egg-info is really a trick for development.  EasyInstall already takes 
care of everything you need for installing as an egg directory, so 
installing .egg-info doesn't really provide any benefit that I see.

Now, if somebody wants to manually set a .pth file (or PYTHONPATH et al) to 
point to a Paste checkout and its included .egg-info, that's fine: just add 
the .egg-info directory to subversion after you've generated it, and you're 
all set.  This is in fact recommended for development scenarios.  You just 
don't want to try and do this by actually *installing* the package and 
.egg-info to site-packages, because then you can't get it back out again.


>  Incidentally, should there be a egg_setup command or something short of 
> creating an egg you won't use to create that special directory?)

I guess I could factor it out into an 'egg_info' command, and call it from 
'bdist_egg'.  It might take some work to prevent duplication, though, and a 
big problem is that it wants to build a 'native_libs.txt' in the info 
directory, which it has to get the information for from the install_lib 
command.  But I could maybe not generate that file in the 'egg_info' 
command, but then there's duplication of process in bdist_egg to find the 
egg-info directory.

No, it doesn't make any sense to do all that extra work.  The idea that the 
egg is something "you won't use" is an illusion in the long run, and 
meanwhile the build process helps validates the correctness of your 
``setup.py`` and much of the rest of your package.  The disk space and CPU 
to build the egg should be minimal, and you only need to do it again if you 
change your package's metadata (e.g. its version).  The rest of the time 
you can use it as-is from subversion or CVS.  So just use bdist_egg.


>Also, if I was just going to get rid of paster install (which is fine, no 
>big investment there), it does have the plugin-enabling code (where 
>plugins can register themselves with Paste).  I'm not sure where that 
>would go; you could put the code into plugins themselves -- very 
>reasonable -- except how do they find the appropriate Paste 
>installation?  I hate PYTHONPATH, and that's where this would lead.

I don't understand what you're trying to do here.  Why do plugins need to 
find Paste?  Shouldn't Paste find the plugins?

My general approach to application plugin installation is that applications 
have a plugin directory where the plugins' eggs get dropped (either via 
EasyInstall, or by the app itself).  The application then just uses 
``find_distributions(directory)`` to iterate over the distributions and 
check for plugin metadata, and do any registering of services provided by 
the plugins.  (Or, you can use the ``AvailableDistributions`` class to 
automatically filter, index, and sort the eggs found.)

For larger application frameworks comparable to Eclipse in the Java world, 
those apps would likely keep a registry of already-installed plugins and 
their registered services, and thus would only scan the plugin directory to 
locate newly- installed plugins.  Some applications might also have an 
explicit installation process of their own.

Anyway, as you can see, there are many architectures you can use to manage 
application plugins using Python Eggs.  I should probably expand on the 
above a bit and add it to the docs, but I'd really like the rest of 
EasyInstall to be done, so that it can use EasyInstall as an example of 
downloading packages and resolving dependencies automatically.  (Although 
in truth most application plugin systems will probably want to just 
download eggs directly rather than building them, and have some 
application-centric directory of download URLs.)


>I almost think it makes sense to just have a hard line for packages that 
>use this stuff: if setuptools isn't installed (or is too old), then go 
>install it and start again.  That way, assuming it is popular, it can 
>become kind of standard regardless of your Python version.

That's my inclination as well.  I suspect that web application deployment 
is a killer use case for eggs, too.  Certainly it was the use case (besides 
Chandler) that pushed me to take action on designing and implementing 
Python Eggs.