[Distutils] Can you collapse easy-install.pth somehow?

[P.J. Eby]

At 12:11 AM 3/4/2009 -0600, skip at pobox.com wrote:
>So I have rather casually installed a bunch of packages from PyPI using
>easy_install.  It now contains 23 eggs, all of which get used a bit, but
>nowhere near as often as the standard lib or our internal stuff.
>Unsuccessfully stat()ing in all those egg directories was really starting to
>hammer our NFS server (we often start up hundreds of Python processes at a
>time), so I commented out the second import line at the end which serves to
>leave the eggs at the end of sys.path.  Now every time I install a package I
>have to quickly edit easy-install.path to remove the import line the install
>process adds.
>
>I wonder though...  Is there some alternate way to install stuff from PyPI
>which reduces the number of directories on sys.path?  Maybe some
>post-processing step which takes the current set of eggs, creates a single
>directory into which they are all unfolded?  That single directory could
>then be added to sys.path.

You could use Ian Bicking's 'pip' installer.  It can't install from 
eggs, but it does do source installs this way.

There is a very straightforward mapping from zipped eggs to "flat" 
files, btw.  For a given .egg file or directory, the mapping is 
simply to unzip it, renaming its EGG-INFO subdirectory to the 
original egg's filename with '-info' appended.  E.g. 
"foobar-1.2-py2.5.egg/EGG-INFO/" becomes 
"foobar-1.2-py2.5.egg-info/".  Everything else remains exactly the same.

It wouldn't be hard to make a tool that does this; in fact, the 
setuptools.archive_util module has an 'extract_archive' function that 
works on directories as well as zipfiles, and can be passed a filter 
function that can change the extraction location for any given 
file.  Something like:


def flatten_egg(egg_filename, extract_dir):
     eggbase = os.path.filename(egg_filename)+'-info'
     def file_filter(src, dst):
         if src.startswith('EGG-INFO/'):
             src = eggbase+s[8:]
             dst = os.path.join(extract_dir, *src.split('/'))
         return dst
     return extract_archive(egg_filename, extract_dir, file_filter)

Of course, this will only work if 'egg_filename' is a valid .egg file 
or directory; the results are otherwise pretty unpredictable.  This 
also doesn't check for overwriting other files, deal with 
permissions, etc.  Ergo, making this production-ready is left as an 
exercise for the implementer.  ;-)