[SciPy-Dev] lazy imports feedback request

[Paul Ivanov]

Friends and uncles,

I'm seeking feedback about some lazy import functionality I'm trying to
implement in nitime. In short, "foo = LazyImport('foo')" returns something that
will act as foo if you did just "import foo" - but only perform the import when
foo is actually used, not when it is defined.

I'll outline the highlights in this email, but you can see the full four-leaf
clover pull request discussion here: https://github.com/nipy/nitime/pull/88

Here's the code, with an explanatory docstring

class LazyImport(object):
    """
    This class takes the module name as a parameter, and acts as a proxy for
    that module, importing it only when the module is used, but effectively
    acting as the module in every other way (including inside IPython with
    respect to introspection and tab completion) with the *exception* of
    reload().

    >>> mlab = LazyImport('matplotlib.mlab')

    No import happens on the above line, until we do something like call an
    mlab method or try to do tab completion or introspection on mlab
in IPython.

    >>> mlab
    <module 'matplotlib.mlab' will be lazily loaded>

    Now the LazyImport will do an actual import, and call the dist function of
    the imported module.

    >>> mlab.dist(1969,2011)
    42.0
    >>> mlab
    <module 'matplotlib.mlab' from '.../site-packages/matplotlib/mlab.pyc'>
    """
    def __init__(self, modname):
        self.__lazyname__= modname
    def __getattribute__(self,x):
        # This method will be called only once
        name = object.__getattribute__(self,'__lazyname__')
        module =__import__(name, fromlist=name.split('.'))
        # Now that we've done the import, cutout the middleman
        class LoadedLazyImport(object):
            __getattribute__ = module.__getattribute__
            __repr__ = module.__repr__
        object.__setattr__(self,'__class__', LoadedLazyImport)
        return module.__getattribute__(x)
    def __repr__(self):
        return "<module '%s' will be lazily loaded>" %\
                object.__getattribute__(self,'__lazyname__')

The win is that when you get a faster import and smaller memory footprint by
not loading those modules which your code path ends up not using.

The functionality of lazyimports.LazyImport is generic enough to allow the
lazily imported module to act as the module in almost every way (tab
completion, introspection for docstrings and sources) except reloading is
not supported.

Although I have not spent a lot of time trying to make reloading work, I think
it is not a big deal because here in the intent is to lazily load external
packages (matplotlib, scipy, and numpy's nosetools) which are not meant to be
developed simultaneously with the project wishing to load them in a lazy
manner.

A difference with another strategy, such as one Keith Goodman highlighted in a
thread announcing Bottlneck 0.4.1 on [SciPy-User] back in March which
looks like this:

# http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_Overhead
email = None
def parse_email():
   global email
   if email is None:
       import email

is that LazyImport keeps you from having to check if a given module
has been lazily
loaded in every place that it is used.

Fernando Perez suggested I contact this list because he recalls similar
functionality being implemented in scipy at some point, and then removed
because it was problematic.

thanks a lot in advance for any feedback. I'm cc-ing my primary
address in hope that your mail reader will do the same.

best,
--
Paul Ivanov