[Distutils] setuptools: namespace packages

[Phillip J. Eby]

At 12:21 PM 8/19/2005 -0500, Ian Bicking wrote:
>When developing a namespace distribution, where should I put it?  Should
>each such distribution be a separate project in the repository (e.g.,
>code goes in Paste/exceptions/trunk/paste/exceptions/)?

If you want to distribute them separately, they need to be separate 
projects with their own setup.py and project name, like "Paste-Exceptions" 
or whatever.

For example, let's say I made "peak.util" a namespace package.  Then, I 
could create separate projects like "PyUUID", "DynamicImports", 
"IndentedStream", and so on, for the different modules in the "peak.util" 
package.  Each one would have to have its own, independently distributable 
project, something like:

     DynamicImports/
         setup.py
         peak/
             __init__.py
             util/
                 __init__.py
                 imports.py

With the setup.py listing 'peak.util' as a namespace package.  The 
__init__.py files will be empty.

The reason you need this layout is that you want "setup.py develop" to 
still work, and for "develop" and "test" to work, they needs the 
__init__.py files.  If you are only using "install" or you always run from 
built eggs, you can forego the __init__.py files and even the directory 
tree, slapping the module directly into the project directory, and 
setuptools will generate stub __init__.py's for you when it builds the 
egg.  (The livinglogic.de libraries like ll-color do this.)

But, if you want to be able to use the "develop" and "test" commands, you 
need a source layout that includes the package tree.

Anyway, the idea behind a namespace package is that you get to claim a 
package name like "paste", and then put all your modules and packages under 
it, without forcing somebody to install a single mega-distribution.  PEAK 
right now has a huge array of functionality under "peak.*" packages, and 
much of it could be used as independent pieces, if only you didn't have to 
install all of PEAK.  Over time (i.e. as soon as I have enough of 
setuptools finished to have some spare time!), I intend to start breaking 
out such pieces as separate distributions, but all the code that still 
imports e.g. "peak.util.imports" will still work, even though it will be in 
a different distribution.

It would be nice, by the way, if somebody who's new to this whole namespace 
package idea would write up an introduction to this for the setuptools 
docs, because the idea is so old hat to me that I doubt I can write a 
really good introduction to it.


>   Can (or should)
>I put them all together and let setuptools break them up when I
>distribute?

Setuptools actually can break them up, using the currently-undocumented 
"feature" facility.  It's undocumented because if you change feature 
options between builds, you may not get what you expect.  But I don't 
really recommend this approach, because then you have to deal with 
--with-X/--without-X options, and then the projects would have to have 
different names, etc.  I went through this thought process when thinking 
about splitting PEAK, as it seemed like it would be easier to leave 
everything together.  But, it's hard to still be compatible with 
EasyInstall that way.  Ultimately I think the "feature" approach is a dead end.



>And will I have to move modules out of the namespace package and into
>subpackages?

No.  A namespace package is a package that's a container.  The container 
can be added to by various distributions; each distribution can add 
modules, resources, or subpackages.  The only thing you shouldn't do is run 
code in the __init__ module, because it will get run as soon as 
pkg_resources is imported and any distribution for the package is on 
sys.path.  For normal site-packages installation, this means that as soon 
as pkg_resources is imported, all declared namespace package __init__.py's 
get run.

If you don't want them to get run right away, you can put this boilerplate 
into all __init__.py files for the package:

     __import__('pkg_resources').declare_namespace(__name__)

and then don't declare the namespace in setup.py.  The problem with this 
approach, however, is that you need to make absolutely certain you do it in 
the mynamespacepackage/__init__.py file of *each* distribution.  If you 
miss even one, and that missed one gets imported first, then the others 
will not get added to the namespace.  This is why it's preferable to use 
the documented approach of listing namespace packages in setup(), and 
always using empty __init__.py files for namespace packages.