[Distutils] API for finding plugins

[Phillip J. Eby]

I recently started work on adding egg support to Chandler ( 
http://chandler.osafoundation.org/ ), and ran into some interesting issues 
with respect to plugin discovery.  Specifically, it's not easy to do it 
well with the APIs that pkg_resources currently offers.  I suspect that 
others who've worked on plugin loading for application environments like 
Zope and Trac have probably run into similar issues.

I'm proposing, therefore, to add a new API to pkg_resources to make 
plugin-finding easier.  Among the requirements:

* It should not cause anything to be imported before it actually needs to 
be used by the application

* It should be able to deal gracefully with multiple installed versions of 
a project in designated plugin directories, falling back to older versions 
if a newer version's requirements can't be met

* It should not actually add anything to sys.path until all plugins have 
been analyzed.

The proposed API would be a 'find_plugins()' method added to the WorkingSet 
class, as follows:

==========

find_plugins(plugin_env, full_env=None)
    Scan `plugin_env` and identify which distributions could be added to 
this working set without version conflicts or missing requirements.

    Example usage::

        distributions, errors = working_set.find_plugins(
            Environment(plugin_dirlist)
        )
        map(working_set.add, distributions)  # add plugins+libs to sys.path
        print "Couldn't load", errors        # display errors

    The `plugin_env` should be an ``Environment`` instance that contains 
only distributions that are in the project's "plugin directory" or 
directories.  The `full_env`, if supplied, should be an ``Environment`` 
instance that contains all usable distributions, *including* those listed 
in `plugin_env`.  If `full_env` is not supplied, one is created 
automatically from the ``WorkingSet`` this method is called on, which will 
typically mean that every directory on ``sys.path`` will be scanned.

    This method returns a 2-tuple: (`distributions`, `error_info`), where 
`distributions` is a list of the distributions found in `plugin_env` that 
were loadable, along with any other distributions that are needed to 
resolve their dependencies.  `error_info` is a dictionary mapping 
unloadable plugin distributions to an exception instance describing the 
error that occurred.  Usually this will be a ``DistributionNotFound`` or 
``VersionConflict`` instance.

    Most applications will use this method mainly on the master 
``working_set`` instance in ``pkg_resources``, and then immediately add the 
returned distributions to the working set so that they are available on 
sys.path.  This will make it possible to find any entry points, and allow 
any other metadata tracking and hooks to be activated.

    The resolution algorithm used by ``find_plugins()`` is as 
follows.  First, the project names of the distributions present in 
`plugin_env` are sorted.  Then, each project's eggs are tried in descending 
version order (i.e., newest version first).  An attempt is made to resolve 
that egg's dependencies.  If the attempt is successful, the egg and its 
dependencies are added to the output list and to a temporary copy of the 
working set.  The process continues with the next project, and no older 
eggs for that project are tried. If the resolution attempt fails, however, 
the error is added to the error dictionary and the next older version of 
the plugin is tried, until a working version is found.

    Note that this algorithm gives precedence to satisfying the 
dependencies of alphabetically prior project names in case of version 
conflicts.  If two projects named "AaronsPlugin" and "ZekesPlugin" both 
need different versions of "TomsLibrary", then "AaronsPlugin" will win and 
"ZekesPlugin" will be disabled due to version conflict.

==========

My question at this point is, is this algorithm sane?  For example, is it 
reasonable to fall back to older versions of plugins in the case of missing 
dependencies or version conflicts, or would it be better to just disable 
those plugins altogether?  I can also imagine that some environments might 
want to have the option of "failing safe" by switching back to a known-good 
prior configuration, or else failing altogether, rather than have arbitrary 
drops or version fallbacks.

Ergo, this API is only a preliminary proposal.  I'd like to hear from folks 
who've tried to implement their own plugin finders, or who would like to 
have one, as to their thoughts on what kind of fallback approaches seem 
best.  That way, I can refine the API better, and perhaps get the benefit 
of some of your field experience with previously-tried approaches.