[Distutils] API for finding plugins

[Phillip J. Eby]

At 11:31 AM 2/7/2006 -0600, Ian Bicking wrote:
>At the same time, I feel like opt-in plugins -- explicitly enumerated in 
>some way -- are going to be necessary for us.  Right now if an egg is not 
>activated by default, you won't see its entry points when scanning by 
>entry point group.  This has confused me before, which I figure is a good 
>predictor that other people are going to be confused by this too.

Others have been confused too, but it's also quite easy to understand once 
you've bumped into it once or twice.  Especially since you really don't 
want the entry points for *every* version of a project to show up in the 
list.  Explicit activation on sys.path is the poor man's global "plugin 
directory", so now I'm finally following up on having a clean way to have a 
*real* plugin directory.


>Right now in Paste there's a couple ways you opt-in.  One is with 
>middleware, which doesn't get picked up implicitly, but instead gets 
>specifically activated and configured.  The other is with PasteScript 
>plugins, where a list of applicable plugins is put in your .egg-info 
>directory for the project.  Those generally work pretty well.  But I'm not 
>sure how to apply that in other areas, like PasteScript commands that 
>aren't project-local.

Configuration files, perhaps?


>I also worry -- but have not actually profiled -- that the scanning is 
>causing Paste Script to be slow on startup (Paste might be too, but I 
>wouldn't notice as much).  It would be good to look into this more 
>closely; but it would also be really good to catch any problems now -- and 
>fix that -- instead of later when it might lead to people bitching about 
>it and talking down setuptools as a result.

FYI, the best way to minimize scan overhead is to avoid repeatedly creating 
Environment instances; this is the slowest thing you can possibly do.  Note 
that most pkg_resources APIs were designed to be called once at startup and 
that's it, so they usually are able to create a default Environment if one 
isn't passed in.  However, if you have an API that gets called "at runtime" 
and it invokes pkg_resources APIs that take Environment instances, you may 
want to create one early and stash it away for future use.  This prevents 
the most expensive scanning from taking place repeatedly.

Note that Distribution objects cache their entry point information, so 
looking for entry points repeatedly on the same WorkingSet doesn't do any 
disk I/O.


>I also want content plugins, like the internationalization plugins (but it 
>might be Javascript or templates or whatever).  And certainly that applies 
>to TG as well.  I've been doing this at work, but using entry points, 
>where the entry points points to an object that is a string that points to 
>a resource that is then fetched with pkg_resources.  Something more direct 
>would be appreciated ;)

Um, you do realize that this is *exactly* what I was campaigning for on the 
Web-SIG when I said, "we need a resource deployment standard", don't 
you?  :)  This is relevant for me for Chandler too; I'd like for us to be 
able to release our coming egg translations and resources library as an 
implementation of a standard, rather than as a Chandler-only thing.  I 
suppose it technically goes beyond the Web-SIG at that point and should 
maybe go to i18n-SIG, or maybe here if distutils/setuptools is part of 
it.  OTOH, web framework and template system authors are among the most 
important stakeholders for such an effort, since they are the most readily 
identified cases of existing systems that would need to implement such a 
standard.

However, it may be that it's a big enough discussion with a diverse enough 
set of stakeholders that it deserves its own (hopefully time limited) 
SIG.  Anyway, I have more cycles that I can devote to a resource deployment 
standard than I can to a templating standard; the former has direct impact 
on my "day job" work, and the latter does not.


>I like entry point groups, and applying the same idea to resources (but 
>without the indirection) would be great.

Again, this is precisely what I wanted to do with a resource deployment 
standard.  (And note also that such a standard would make a more 
standardized approach to the find_template issue much more 
possible/practical -- another reason I thought we should tackle deployment 
before rendering.)


>Hmm... and reading over this, I realize that a "plugin directory" is not 
>that different from the virtual python setups, and I think that kind of 
>setup is really the only good way to deploy web applications.  "Working 
>set" -- which is already an idea in setuptools -- is probably the concept 
>that encompasses both uses.  As a developer, I'd like it to be easier and 
>safer to change my working set -- right now I have hacks in sitecustomize 
>and distutils to make that easy.  This switching of working sets is 
>basically what Jim wants in a Zope Instance as well.

Ideas welcome, especially ones with working implementations.  ;)