Module import hook

[Thomas Calmant]

Hi,

For the context, I'm working on Pelix (https://github.com/tcalmant/ipopo),
a service-oriented architecture framework (in GPLv3), inspired by OSGi
(from the Java world).
It runs on Python >= 2.6 (with the backport of importlib) and Python 3.1
(not tested upon this version).

It considers Python modules as "bundles", i.e. artifacts having a
life-cycle, that can be installed, started, updated, stopped and
uninstalled.

The current implementation allows to install each module individually
according to their full name, using the importlib standard module.
I'm not yet updating the sys.modules dictionary, so any import made in
module will use a different instance than the one created by an
installation using the framework (if any).

I've made some tests the code is available here:
https://dl.dropboxusercontent.com/u/59622687/ModuleLoaderTest.tar.gz

1/ I'd like have the equivalent of a distinct sys.modules dictionary per
framework instance, in order to solve the problem of the different versions
of an imported and installed module, in the same framework.

In my tries, I used a DictView class that replaces sys.modules during the
import process, which allows to write in a given dictionary, but allows to
read from other ones too.
This way, the import process can write in a framework dictionary and can
see the rest of the original sys.modules too (which is left untouched).

To hook the imports:
- I've tried overriding __import__, but I'm not comfortable with it. The
replacement of sys.modules doesn't seem to be enough in Python 2 in this
case.
- I've tried with the Finder/Loader mechanism, but I didn't succeed in
having the exact same behavior than without hooks: I get recursive loops in
some cases in Python 2, or can't find some sub-modules in Python 3.
Also, my finder/loader are based on the imp.find/load_module method, which
will be deprecated in Python 3.4.

a) Which mechanism would you use to implement such a behavior ?
b) Is the deprecation of imp methods a big problem ?
c) Is there any clean Finder/Loader implementation that acts like the
standard import process ? All those I've found on the Internet had the same
problems than me.

Also, I'd like to keep track of the imports from a module installed in the
framework, to store them inside the framework modules dictionary (which is
an extension of point 1, I suppose), even if they are late imports, e.g.
from inside a method.
I'm nearly OK with that using the Finder/Loader mechanism.

As all I want is being called before and after every import (to set up
sys.modules and the framework, then clean it up), without re-implementing
the searching and loading of modules,
is there a way to be only notified of those events ?


2/ I'd like to have a method to install modules by file path, and to
install the modules present in a package folder (using the visitor pattern).
Once again, I think using the imp.find/load_module method to handle a
module file path.
For the package loading, the implementation in loader.py is based on
pkgutil.iter_modules.

This part seems to work, but I'm facing the bugs described in the first
point.
Once again, which mechanisms would you use to do that ?


Thanks for your answers :)
Regards,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20130416/c5b0fadc/attachment.html>