[Distutils] Splitting large packages into multiple eggs

[Robert Kern]

Bob Ippolito wrote:

> The issue at hand is how to structure the setup.py to support  
> creation of multiple eggs, with an egg for installation purposes that  
> depends on everything.  PyObjC can be broken up into about 30 eggs,  
> one for each package, one for the Xcode support (which depends on  
> py2app and altgraph), one for all of the tests (or maybe separate  
> eggs for each test suite).  Obviously I'm not looking to create 30+  
> setup.py files, so what do I do?

You may want to look at how the new scipy.distutils scheme works. We have a
utility class Configuration which encapsulates everything about the setup.
Eventually, it creates the **kwds for setup() from that information. The
important bits of our main setup.py looks something like this:

  from scipy.distutils.misc_util import Configuration
  config = Configuration(
                         maintainer = "SciPy Developers",
                         # ...
                         )
  # Force scipy to be a package (its __init__.py file comes from scipy_core)
  config.packages.append('scipy')
  config.package_dir['scipy'] = os.path.join(config.local_path,'Lib')

  config.add_subpackage('Lib')

  setup(**config.todict())

In this case, the Lib/ subdirectory in the source tree will become the main
scipy package. It has its own setup.py:

  def configuration(parent_package='',top_path=None):
    from scipy.distutils.misc_util import Configuration
    config = Configuration('scipy',parent_package,top_path)
    config.add_subpackage('sandbox')
    config.add_subpackage('utils')
    config.add_subpackage('io')
    config.add_subpackage('fftpack')
    config.add_subpackage('signal')
    config.add_subpackage('integrate')
    config.add_subpackage('linalg')
    config.add_subpackage('special')
    config.add_subpackage('optimize')
    config.add_subpackage('stats')
    config.add_subpackage('interpolate')
    config.add_subpackage('sparse')
    config.add_subpackage('cluster')
    config.add_subpackage('lib')
    config.make_svn_version_py()  # installs __svn_version__.py
    config.make_config_py('__scipy_config__')
    return config

Each of these subdirectories has their own setup.py files, too. E.g.:

  def configuration(parent_package='',top_path=None):
    from scipy.distutils.misc_util import Configuration
    config = Configuration('io', parent_package, top_path)
    config.add_extension('numpyio',
                         sources = ['numpyiomodule.c'])
    config.add_data_dir('tests')
    config.add_data_dir('examples')
    config.add_data_dir('docs')
    return config

All of these file names and directories are local. I could move subpackages from
scipy.sandbox.*, say, into scipy.*, and all I'd have to change are the
config.add_subpackage() calls. The Configuration class does all of the bookkeeping.

One could selectively build certain subpackages. One of these days I am going to
write a script that will read data from a configuration file to determine which
subpackages to build.

The Configuration class seems to be fairly decoupled from the rest of
scipy.distutils. I think you only have to change a few

  from scipy.distutils.core import Extension

to

  from setuptools import Extension

http://svn.scipy.org/svn/scipy_core/trunk/scipy/distutils/misc_util.py

-- 
Robert Kern
robert.kern at gmail.com

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter