[Python-Dev] Dynamic module namspaces

[Andrew Bennetts]

On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote:
> In an effort to reduce the memory usage used by GTK+ applications 
> written in python I've recently added a feature that allows attributes 
> to be lazy loaded in a module namespace. The gtk python module contains 
> quite a few attributes (around 850) of which many are classes or 
> interfaces (150+)

Have you seen the "demandload" hack that Mercurial uses?  You can find it here:
    http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py

You can see an example use of it here:
    http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py

The advantage for an interactive command line tool isn't so much memory
consumption as speed.  Why waste hundreds of milliseconds importing code that
isn't used?  There's an experimental branch to use the same demandload code in
bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root'
from 400ms => 200ms, etc." (according to
http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967)

Over half the runtime wasted on importing unused code!  There's a definite need
for a nice solution to this, and it should be included in the standard
batteries that come with Python.

If we can address related problems at the same time, like emitting deprecation
warnings for accessing certain module attributes, then even better!

-Andrew.