Developing modules with ‘pkgutil’

[Ben Finney]

Howdy all,

With all the current fuss about namespace packages, I'm dealing with a
similar goal that I hope is possible with stock Python 2.x.

When developing a module intended to be part of an existing package
installed on the system, I want to write programs that will import the
*in-development* module from my current working tree, but the
*system-installed* modules from the rest of the package.

The specific case is that of a new writer for Docutils. The module will
eventually be installed such that it is accessed via ‘import
docutils.writers.foo’, but while developing it I explicitly *don't* want
the module to come from the system-installed ‘docutils/writers/’
directory since that will fail to get the one I'm developing.

But the rest of the Docutils infrastructure I *need* to come from the
system-installed package when I do ‘docutils.bar.baz’. So I need to
somehow have a module in one location appear, for the purpose of
imports, as though it's part of a package defined in a different
location.

A naive first try would be:

    /usr/lib/python2.5/site-packages/
        docutils/
            __init__.py
            core.py
            …
            readers/
                __init__.py
                …
            transforms/
                __init__.py
                …
            writers/
                __init__.py
                …

    $HOME/projects/foo-writer/
        tools/
            rst2foo
        docutils/
            __init__.py
            writers/
                __init__.py
                foo.py

So, while developing the ‘foo.py’ module, I need to run ‘tools/rst2foo’
in order to make use of the module and exercise its in-development
functionality (and *not* whatever ‘foo.py’ might exist in the system
library location). But whenever I import anything *else* from the
‘docutils’ package, it's not going to find the system-installed package.

The ‘pkgutil’ module <URL:http://docs.python.org/library/pkgutil>, which
I only partly understand (with thanks to Doug Hellman for
<URL:http://blog.doughellmann.com/2008/02/pymotw-pkgutil.html>), appears
to offer a solution for this, by allowing my in-development ‘docutils’
shim package to also include the real one:

    $ cd $HOME/projects/foo-writer/
    $ cat docutils/__init__.py
    import pkgutil
    __path__ = pkgutil.extend_path(__path__, __name__)
    __path__.reverse()

Which, according to Doug, should allow me to have ‘tools/rst2foo’ say:

    from docutils.core import publish_cmdline, default_description
    from docutils.writers import foo

as it will look when installed for the end user; and the ‘docutils’
namespace will, because of the in-development shim package, be searched
both there and the system-installed directory, finding both the
system-installed ‘docutils.core’ and the in-development
‘docutils.writers.manpage’.

Then I should be able to run the program while developing, like this:

    $ PYTHONPATH=.  python ./tools/rst2foo

and it should find everything it needs.


However, this approach doesn't seem to be complete: there is code in the
system-installed ‘docutils/__init__.py’ that is not being found:

    Traceback (most recent call last):
      File "./tools/rst2foo", line 21, in <module>
        from docutils.core import publish_cmdline, default_description
      File "/usr/lib/python2.5/site-packages/docutils/core.py", line 20, in <module>
        from docutils import __version__, __version_details__, SettingsSpec
    ImportError: cannot import name __version__

What's happening, of course, is that the name ‘__version__’ is defined
in the system-installed ‘docutils/__init__.py’, but that's not being
found because the in-development ‘docutils/__init__.py’ is found first,
which doesn't have that name.

A simple ‘from docutils import *’ clearly doesn't work, for the same
reason. Catch-22.

At this point I'm stuck. I can't see how to have the
‘docutils/__init__.py’ stop shadowing the names in the system-installed
‘docutils/__init__.py’, while still doing the namespace shuffle
necessary to have my in-development module appear part of the wider
package namespace. What should I be doing instead?

-- 
 \         “Smoking cures weight problems. Eventually.” —Steven Wright |
  `\                                                                   |
_o__)                                                                  |
Ben Finney