[Distutils] slow Python startup with a couple of easy_install-ed packages

[Phillip J. Eby]

At 08:44 PM 2/23/2007 +0100, Thomas Heller wrote:
>Well, easy_install makes it snap to install packages that I may or may
>not use. Nothing is easier than to 'easy_install 
>something-recently-announced',
>even if I never will use it (after a preliminary look).  Often I forget to
>uninstall the packages again, or promise to do that later. Disk space is 
>cheap.
>
>However, I found that with the ~20 eggs that I have installed the startup time
>of the Python interpreter (Python2.5, WinXP, measured with 'timeit python 
>-c ""')
>rises considerably - 2.5 times slower.
>
>This only occurs when the eggs are listed in the easy_install.pth file; if 
>they
>are moved into a 'normal' pth file this slowdown does not occur, although 
>I can
>still import the eggs fine.
>
>I assume the additional time is spent by setuptools looking up entry points or
>whatever?

No.  What's happening is that easy-install.pth puts the eggs at the 
*beginning* of sys.path, in order to ensure that eggs override non-egg 
packages; this prevents a *huge* amount of complication in sys.path 
management with respect to detecting conflicts, as well as allowing eggs to 
be installed that override the standard library (e.g. to allow hotfixing it).

The downside is that this means eggs are searched before the stdlib, 
slowing down the import of stdlib modules at startup.

The easy fix is to use the "-m" option, which doesn't put the eggs in any 
.pth file at all -- and removes them if they're there.


>IMO this is a critical problem of easy_install...  (Imaging how the 
>installed and
>possibly never used eggs pile up).

How many of these eggs are directories, vs. how many are 
zipfiles?  Zipfiles shouldn't be having so much effect.

Meanwhile, in setuptools 0.7 there will be a completely different way of 
managing installation locations, and it will not depend on .pth files or on 
putting eggs first.  Instead, it will use direct old-style installation, 
combined with .egg-info directories for metadata and for installation 
manifests to support automated uninstallation, upgrades, and integrity 
verification.