Pyitect - Plugin architectural system for Python 3.0+ (feedback?)

[Ben Powers]

on Tue, Jun 16,  2015 at 17:49 Ian Kelly <ian.g.kelly at gmail.com> wrote

>On Mon, Jun 8, 2015 at 10:42 PM, Ben Powers <ryexander at gmail.com> wrote:
>> As importlib has been added in python 3 and up I decided to use it's
>> abilities to create a plugin system for truly modular development in python.
>>
>> Pyitect has the ability to drop in components and resolve dependencies. Even
>> load different versions of a dependency if two different libraries require
>> two different versions.
>>
>> Pyitect gives each "plugin" it won private namespace to import the
>> components it needs. and nothing else.
>>
>> Pyitect  organizes it self around the concept of a System. Each system can
>> scan and load it's own set of plugins, in each system plugins can be enabled
>> individually.
>>
>> Pyitect even comes with a very simple event system that it uses internal to
>> fire event when plugins are found/loaded/ect for the developer to use.
>>
>> I've written up What I think is a quite comprehensive readme to go with it.
>>
>> Available on PyPi https://pypi.python.org/pypi?:action=display&name=pyitect
>> and Github https://github.com/Ryex/pyitect
>>
>> I'm looking for feedback on what flaws the system may have and where I can
>> improve. I'm always open to pull requests
>>
>> Looking forward to what the comunity thinks of this little project of mine.
>> Thanks for your input and criticism.>

>You need to run your readme through a spell-check, as it's riddled
>with typographical errors and quite hard to read. A grammar check and
>a proofreader would also be useful.>


Actually ya, looking through it again there are number that I didn't find

>> "on_enable": "dostuff.py", // optional, runs this file when the plugin is enabled>

>A path to a function would feel more Pythonic.>


You right that WOULD be more pythonic, Why I didn't think of that at
the time is a compleat mistery to me.


>> mode -> (OPTIONAL) defaults to import on python 3.4 and up ecec otherwise: sets the import mode>

>Do the plugins need to be on the sys.path in order to be imported?>


No. plugins need only be refered to by a path relative to their .json file.

>> provides -> a mapping of provided component names to prefix mappings>

>What are the "prefix mappings" and what are they used for?>

>On Mon, Jun 8, 2015 at 10:42 PM, Ben Powers <ryexander at gmail.com> wrote:
>> As importlib has been added in python 3 and up I decided to use it's
>> abilities to create a plugin system for truly modular development in python.
>>
>> Pyitect has the ability to drop in components and resolve dependencies. Even
>> load different versions of a dependency if two different libraries require
>> two different versions.
>>
>> Pyitect gives each "plugin" it won private namespace to import the
>> components it needs. and nothing else.
>>
>> Pyitect  organizes it self around the concept of a System. Each system can
>> scan and load it's own set of plugins, in each system plugins can be enabled
>> individually.
>>
>> Pyitect even comes with a very simple event system that it uses internal to
>> fire event when plugins are found/loaded/ect for the developer to use.
>>
>> I've written up What I think is a quite comprehensive readme to go with it.
>>
>> Available on PyPi https://pypi.python.org/pypi?:action=display&name=pyitect
>> and Github https://github.com/Ryex/pyitect
>>
>> I'm looking for feedback on what flaws the system may have and where I can
>> improve. I'm always open to pull requests
>>
>> Looking forward to what the comunity thinks of this little project of mine.
>> Thanks for your input and criticism.>

>You need to run your readme through a spell-check, as it's riddled
>with typographical errors and quite hard to read. A grammar check and
>a proofreader would also be useful.>

Actually ya, looking through it again there are number that I didn't find

>> "on_enable": "dostuff.py", // optional, runs this file when the plugin is enabled>

>A path to a function would feel more Pythonic.>

You right that WOULD be more pythonic, Why I didn't think of that at
the time is a compleat mistery to me.

>> mode -> (OPTIONAL) defaults to import on python 3.4 and up ecec otherwise: sets the import mode>

>Do the plugins need to be on the sys.path in order to be imported?>


No. plugins need only be refered to by a path relative to their .json file.

>> provides -> a mapping of provided component names to prefix mappings>

>What are the "prefix mappings" and what are they used for?>

That should actually read "postfix mappings" my bad.

normally when you list the components a plugin provides the names of
the components are assumed to exist in the plugin's module name space,
and when the components are required pyitect simply pulls the objects
by name out of the module and passes them along.

postfix mappings allow components to be named differently in the
module than they are provided and allows the developer to attach
version postfixes to the component.

an example from the README:
{
    "name": "Im-A-Plugin",
    ...
    "version": "0.0.1",
    ...
    "provides": {
        "Bar": "bar_type_1=bar1"
    }
}

This provides the "Bar" component at version 0.0.1-bar_type_1 by
pulling the name `bar1` from the imported plugin module.
the the version postfix string was "=bar1" then the postfix the the
version would be removed just 0.0.1 it would still import under `bar1`
instead of `Bar`

>> #file.py
>> from PyitectConsumes import foo
>>
>> class Bar(object):
>>     def __init__():
>>         foo("it's a good day to be a plugin")>

>I don't understand this example. Is "PyitectConsumes" the name of a
>component or part of Pyitect? Or is "foo" the name of the component?>


`PyitectComsumes` is a special module injected into sys.modules when
the plugin module is imported and removed after the import is
complete. it's name-space contains all the components declared as
required in the plugin's .json


>> plugins = [system.plugins[n][v] for n in system.plugins for v in system.plugins[n]]>

>What are n and v?>

`n` and `v` are short variable names for `name` and 'version'

system.plugins is a mapping of plugin names to a mapping of plugin
versions to plugin objects.

ie:

{
	"Plugin_Name": {
	    "0.0.1": <pyitect.Plugin Object>
	}
}

this is working python code that pulls all the plugin objects in the
system into an array, potentially to be enabled.
filtering could then be done when creating this array or after if it
needed to prevent plugins from being enabled or the list could be
constructed manually from a white list.

plugins can be searched for and or enabled at anytime in the plugin
system life cycle but the components they provide are not availably
until they are enabled.


>> Plugins are loaded on demand when a component is loaded via
>>
>> System.load("<component name>")>

>What's the difference between this and the "PyitectConsumes" import?>


The PyitectConsumes import is for use inside plugins where
dependencies are declared prior to import.

System.load is for use in the application to load components. Plugins
are not intended to know about the System they are being used in.


>> ### both in both cases relative imports DO NOT WORK. the plugin folder is temporarily added to the search path so absolute imports work but relatives will not.
>>
>> UNLESS the name of the file is __init__.py . In this special case the plugin folder is reconsidered as a python package and relative imports work as normal.>

>Isn't this just how relative imports normally work? Of course you
>can't use a relative import outside of a package.>


This is true, however some might make the mistake of assuming that
plugins are treated like packages even they do not include a
__init__.py file in the plugin's folder when really they are treated
as any python module.


>> Loaded pluginss do NOT store their module object in sys.modules>

>What about imports of plugin-local modules that are performed inside
>the plugins? Do those also get cleaned up from sys.modules?>


Actually no... thats an oversight on my part. really I should
temporally clone sys.modules inject the PyitectConsumes special module
and then set it back to the original sys.modules dict after the import
has been performed, that would secure the process more and prevent
cluttering of sys.modules.

thank you for pointing this out.

>If I have version 1 of a plugin that imports a module from its
>package, and version 2 of the same plugin that imports a module from
>its package, are those imports going to conflict with one another?>

when the plugin's are enabled their components are mapped into the
system. the first time a component is required and a compatible
version to the request is available a fresh module object with a blank
namespace is constructed and the file declared in the .json is
imported/executed (via `compile` ing a code object then `exec` acuting
the code object) with the fresh plugin obj's namespace.

as the only way to have to plugins with different version is the have
two different folders and given my understanding of how python
searches for modules to import I do believe it is impossible for there
to be conflict with imports internal to a plugin.



>Also, earlier you appeared to instantiate the System class, but now
>you appear to be just accessing attributes from the class directly.
>What's the distinction between the class-level state and the
>instance-level state?>


The class level state does not exist. that I suppose is a
miscommunication of the README. to use a plugin system you must
instantiate a System object and then populate it with plugins to use,
then enable the plugins to make their provided components available
for use.


>> System.useing
>> System.enabeled_plugins

>Are these misspellings actually part of the API?>

Yes and no, to my shame System.useing instead of System.using exists
but System.enabled_plugins is correctly spelled in the API

>Also, earlier you appeared to instantiate the System class, but now
>you appear to be just accessing attributes from the class directly.
>What's the distinction between the class-level state and the
>instance-level state?>

The class level state does not exist. that I suppose is a
miscommunication of the README. to use a plugin system you must
instantiate a System object and then populate it with plugins to use,
then enable the plugins to make their provided components available
for use.

>> Providing multiple versions of a component from the same plugin>

>Is there a reason to do this as opposed to simply providing multiple components?

Yes actually.

imagine the case where you have a set of data to which there are many
operation that could be done. you are going to operate on this data
through a interface.
lets say you dub components that implement this interface "tools". you
want plugins to provide tools to work with the data in different ways.
So, you use a "Tool" component. Different versions of the "Tool"
component in this case can be assumed to be different tools.
If someone wants to provide several tools but only wants to write one
plugin then they would need to provide different version of the same
component from the same plugin.

The application could then enumerate over the different versions of
the component available, perhaps filtering out those that appear to be
different version of the same tool if there was some convention
applied to the version string (ie. including the tool name in the
version string 0.0.1-tool-x" (postfix mapping?))
and call on each tool as necessary

There are many possible use cases, the point was to make the system
flexible. Perhaps there is a better way of achieving this goal but
thats what I came up with.


Hopefully I answer your questions to your satisfaction, thanks for the feedback.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150622/7ec2cf74/attachment.html>