[Distutils] Performance implications of having large numbers of eggs?

[Jim Fulton]

On Jun 28, 2007, at 12:05 PM, Rick Ratzel wrote:

>
>>    Date: Thu, 28 Jun 2007 10:47:43 -0500
>>    From: Dave Peterson <dpeterson at enthought.com>
>>
>>    Has anyone done any investigation into the performance  
>> implications of
>>    having large numbers of eggs installed?  Is there any sort of
>>    performance hit?
>>
>>
>>    It seems to me that having a really large path might slow down  
>> imports a
>>    bit, though I suspect this is in C code so probably not a  
>> significant
>>    problem.  It also seems like there might be some startup  
>> penalties due
>>    to the overhead of setting up the path when using eggs, but  
>> this is a
>>    one-time cost during python startup, so probably not too bad  
>> either.
>
>    Another option to avoid a startup penalty is to have all eggs  
> installed with
> -m (not in the easy-install.pth file, or "deactivated") and have  
> code require()
> the specific dependency.  This has the obvious disadvantage of  
> having to change
> a fair amount of code.  But, the advantages are only adding the  
> eggs that are
> needed to the path when they're needed (instead of every egg in  
> every PYTHONPATH
> dir), and your code will be sure that it's using the version that it's
> compatible with.
>
>    I should mention that I don't have any metrics on the startup  
> penalty, so a
> change like this may not be worth it if you're only trying to  
> improve that.

Note that buildout takes this a step further by determining what eggs  
are needed at install time, rather than run time.  I imagine that  
this could speed startup further, but I don't have any metrics  
either. :)  Of course, you could use the same approach without using  
buildout.  Note that sooner or later, I'm pretty sure we're going to  
need a more clever algorithm, likely with some sort of backtracking,  
to determining working sets. At that point, it will become very  
important to not do this at run time.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org