Pyitect - Plugin architectural system for Python 3.0+ (feedback?)

[ryexander at gmail.com]

On Monday, June 22, 2015 at 1:18:08 PM UTC-6, Ian wrote:
> On Mon, Jun 22, 2015 at 2:32 AM, Ben Powers <ryexander at gmail.com> wrote:
> > on Tue, Jun 16,  2015 at 17:49 Ian Kelly <ian.g.kelly at gmail.com> wrote
> >
> >>On Mon, Jun 8, 2015 at 10:42 PM, Ben Powers <ryexander at gmail.com> wrote:
> >>> #file.py
> >>> from PyitectConsumes import foo
> >>>
> >>> class Bar(object):
> >>>     def __init__():
> >>>         foo("it's a good day to be a plugin")>
> >
> >>I don't understand this example. Is "PyitectConsumes" the name of a
> >>component or part of Pyitect? Or is "foo" the name of the component?>
> >
> >
> > `PyitectComsumes` is a special module injected into sys.modules when the
> > plugin module is imported and removed after the import is complete. it's
> > name-space contains all the components declared as required in the plugin's
> > .json
> 
> So this import can only be performed when the module is loaded, not
> later when a component's functionality is being invoked?
> 
> >>> Plugins are loaded on demand when a component is loaded via
> >>>
> >>> System.load("<component name>")>
> >
> >>What's the difference between this and the "PyitectConsumes" import?>
> >
> >
> > The PyitectConsumes import is for use inside plugins where dependencies are
> > declared prior to import.
> >
> > System.load is for use in the application to load components. Plugins are
> > not intended to know about the System they are being used in.
> >
> >
> >>> Loaded pluginss do NOT store their module object in sys.modules>
> >
> >>What about imports of plugin-local modules that are performed inside
> >>the plugins? Do those also get cleaned up from sys.modules?>
> >
> >
> > Actually no... thats an oversight on my part. really I should temporally
> > clone sys.modules inject the PyitectConsumes special module and then set it
> > back to the original sys.modules dict after the import has been performed,
> > that would secure the process more and prevent cluttering of sys.modules.
> 
> It seems to me that this could create new problems. For example,
> suppose a plugin being loaded imports urllib, and then later the
> application tries to import urllib. It's not in sys.modules, so the
> import framework loads it again, and now there are two copies of
> urllib floating around in the interpreter. I think you actually need
> to diff sys.modules and then remove anything that's added but only if
> its path falls within the path of the plugin directory.
> 
> The more that I think about this, I don't think that overloading the
> import machinery like this is the right way for plugins to gain access
> to components. If instead you just pass the module a context object
> with a get_component method, then you won't have to muck around with
> sys.modules as much, and the context object can remain available to
> the plugin for later use.

well I've done some reworking of the system trying to take the points you've brought up into account. your probably right this would be best done without mucking with the import machinery. passing a module a context object as you say would be a much preferred method. unless were using direct code execution on a module object's namespace like so

filepath = os.path.join(self.path, self.file)
module_name = os.path.splitext(self.file)[0]
# exec mode requieres the file to be raw python
package = False
if module_name == '__init__':
    module_name = os.path.basename(self.path)
    package = True
try:
    plugin = types.ModuleType(module_name)
    if package:
        plugin.__package__ = module_name
        plugin.__path__ = [self.path]
        sys.modules[module_name] = plugin
    else:
        plugin.__package__ = None
    sys.path.insert(0, self.path)
    with open(filepath) as f:
        code = compile(f.read(), filepath, 'exec')
        exec(code, plugin.__dict__)
    sys.path.remove(self.path)
    if package:
        del sys.modules[module_name]

there is no way to make this context object available during the module load as importlib offer no functionality to provide a context during import.

and if the context object WAS provided this way by injecting it into the module object's __dict__ before the plugin code was executed it would exist as a magic object with no formal definition you just have to trust that it exists.

Also short of some convoluted declaration of globals in the plugin modules namespace, there is no way to pass in the plugin system so that the plugin can acquire components on demand short of the author of the plugin system author doing it themself by explicitly passing in an instance of the system.

Ultimately I think it best to inject a special Module into sys.modules during the import and require the plugin author to pull anything they might need during load from that. if they need to save it for later there is always

import PyitectConsumes as foobar

then use foobar.componentName there after

I think it far more pythonic to fail for lack of dependency during import of the plugin than during execution

If the plugin author needs to do things on demand then the system author will need to make available a way to access the plugin system instance.

I HAVE implemented the on_enable via  providing a path to a callable in the imported module, that was a much better way than specifying a separate file.

as to your point about plugin import like urllib getting duplicated if I implimented the sys.modules clone to keep it form getting cluttered... I not entirely sure this would be a bad thing. It would certinly lead to higher memory usage, depending on the dubed module's implementation it might cause undefined behavior... but I can think of a few way around that namely pre importing such modules before the plugin so that they would be in sys.modules before it gets cloned.
I guess it comes down to the decision of "do I let sys.moudles get cluttered? or do I force plugin authors to declare external imports to avoided duplicate imports?" honestly I can't decide there.