[Python-Dev] Dynamic module namspaces

[Phillip J. Eby]

At 03:38 PM 7/15/2006 -0300, Johan Dahlin wrote:
>In an effort to reduce the memory usage used by GTK+ applications
>written in python I've recently added a feature that allows attributes
>to be lazy loaded in a module namespace. The gtk python module contains
>quite a few attributes (around 850) of which many are classes or
>interfaces (150+)
>
>The changes to PyGTK I had to make can not be considered anything but a
>hack; I had to put a subclass of a ModuleType in sys.modules and
>override __getattribute__ to be able to get old code which accessed
>gtk.__dict__ directly to still work (PyModule_GetDict requires that).

Just as a point of reference, the Importing package does something very 
similar, to support "weak" and "lazy" imports:

http://cheeseshop.python.org/pypi/Importing


>However, even if I didn't have to use __getattribute__ overriding
>sys.modules is rather unpleasent and I'm afraid it'll cause problems in
>the future.

The things most likely to be problems are tools like pydoc or other 
inspect-using code that expects modules to be exactly ModuleType and don't 
use isinstance().  Apart from that, I've been using the technique since the 
early days of Python 2.2 without encountering any problems until the PEP 
302 "reload()" bug came along, but that was fixed in 2.3.5.  I haven't seen 
any other problems since.

On the other hand, the Importing package takes a much more conservative 
approach than you are doing; it simply runs reload() on a module when 
__getattribute__ is called (after restoring the old version of 
__getattribute__).  So, as soon as you touch a lazily loaded module, it 
ceases to be particularly special, and it has a real __dict__.  It's 
possible that what you're doing could have more side-effects than what I'm 
doing.


>What I want to ask, is it possible to have a sanctioned way to implement
>a dynamic module/namespace in python?

That would be nice, but I think that what you and I are doing are probably 
the One Obvious Ways to do the respective things we're doing.