[SciPy-dev] new buld process in CVS

[eric]

SciPy's build process has been revamped.  Thanks to Pearu for tons of help
in this area on
getting this all setup.  Please grab the CVS and test on your given
architecture so we can
find out what has broken.  Please report both success and failure.

You'll need the latest snapshot of f2py (see below).  Also, you'll need to
set
the following line in setup.py to point to your atlas libraries:

    scipy_distutils.atlas_info.library_path = ['home/ej/lib/atlas']

Then:

    python setup.py build
    python setup.py install
    python
    >>> import scipy
    >>> scipy.test()

The main purpose of the reorg was (1) to modularize the build process so
that
it is easier to maintain, (2) make it easy for others to build packages that
can be dropped into SciPy with minimum fuss, and (3) Allow for building
and installing individual modules outside of SciPy without changing any
install
scripts. There are notes about how to do (2) at the bottom of this file.

thanks,
eric

new dependencies:

    f2py2e (http://cens.ioc.ee/projects/f2py2e/)

        The latest snap shot is needed and is here:
            http://cens.ioc.ee/projects/f2py2e/rel-5.x/
        This must be installed to build SciPy -- it has a setup.py script so
it is
        easy.

old dependencies:  You still need these.

    atlas
    fftw

changes:

    * scipy_distutils is a new package (installed by scipy, but as a
separate package)
      that provides support for Fortran, f2py, and most of the other
customizations
      that scipy uses for distutils.  It is also used by Pearu's f2py
package, and
      he's made a ton of contributions here.

    * setup.py is now very compact. Most setup work is delegated to
sub-packages.

    * each sub-package has a setup_foo.py file in it that has a
configuration()
      method.  This returns a dictionary of key word arguments that can be
passed
      to setup.py to build the package.

      A simple example can be found in fastumath/setup_fastumath.py.  More
      complex examples are xplt/setup_xplt.py
      and linalg/setup_linalg.py.

    * scipy_distutils.atlas_info attempts to find your atlas library
installation.
      It is pretty dumb right now.  It has a variable called library_path
that you
      can use to set the path to your atlas directory.  Note that it takes a
list,
      not a string as the argument:

        import scipy_distutils.atlas_info
        scipy_distutils.atlas_info.library_path = ['/home/ej/lib/atlas']

      fftw_info.py serves the same purpose for fftw.  Still need to add
      library_path to it.

    * scipy.compiler has been renamed to scipy.weave.  It has been
significantly
      enhanced in its capability.  (still a few bugs here though.)

    * Numeric is no longer installed by the setup.py script.  You'll have to
install
      it separately (we'll continue to include it in the windows executables
at
      point releases)

    * fastumath is installed as a separate module instead of being dropped
in the
      Numeric directory.

    * scipy_test.py is now a package (do to distutils issues) and installed
as a
      separate stand-alone package.

    * SciPy now builds with compilers other than g77 on Linux. Not much
testing
      has been done yet with these:

        * Intel Fortran (by Pearu Peterson)
        * Absoft 5.0 (compiles, but still problems with upper/lowercase and
                      underscore issues)

------------------

To include the package foo within SciPy, you'll need to place the foo
directory
within the SciPy directory, and create a setup_foo.py script within that
directory:

    scipy/
        foo/
            foo1_source.c
            foo2_source.c
            __init__.py
            foo_script.py
            setup_foo.py

setup_foo.py should have a single method in it called configuration() that
looks
like:

    #setup_foo.py
    import os
    from scipy_distutils.core import Extension
    from scipy_distutils.misc_util import get_path, default_config_dict

    def configuration(parent_package=''):
        # if we have a parent module, append a '.' to it
        # to simplify adding it to package names, etc.
        if parent_package:
            parent_package += '.'

        # find the local path of this module (/home/eric/scipy/foo)
        local_path = get_path(__name__)

        # grab a default configuration dictionary.  We'll fill this
        # with this packages specifics.
        config = default_config_dict()

        # Set the name of this package.
        config['packages'].append(parent_package+'foo')

        # build the first extension module
        sources = ['foo1_source.c']
        sources = [os.path.join(local_path,x) for x in sources]
        ext = Extension(parent_package+'foo.foo1',sources)
        config['ext_modules'].append(ext)

        # build the second extension module
        sources = ['foo2_source.c']
        sources = [os.path.join(local_path,x) for x in sources]
        ext = Extension(parent_package+'foo.foo2',sources)
        config['ext_modules'].append(ext)

        # return the configuration information.
        return config

Second, you have to add the name of the module 'foo' to one of the
installation
lists in setup.py.

    # setup.py (in scipy directory)
    .
    .
    .
    # standard modules
    standard_packages = ['io','linalg','special','signal','stats',
                         'interpolate','integrate','optimize',
                         'cluster','cow','ga','weave',]
    .
    .
    .

That is about it.  The build process should detect your package and install
it
as a sub-package of scipy.  If you'd like to bundle your package with SciPy,
but
want to have it installed as a stand-alone package, put it in the following
list of packages:

    # setup.py (in scipy directory)
    .
    .
    .
    # these packages aren't nested under scipy
    separate_packages = ['gui_thread','scipy_test','scipy_distutils',
                             'fastumath']
    .
    .
    .


The default_config_dict function looks like this:

    # scipy_distuils/misc_util.py
    .
    .
    .
    list_keys = ['packages', 'ext_modules', 'data_files',
                 'include_dirs', 'libraries', 'fortran_libraries',
                 'headers']
    dict_keys = ['package_dir']

    def default_config_dict():
        d={}
        for key in list_keys: d[key] = []
        for key in dict_keys: d[key] = {}
        return d
    .
    .
    .

Let me know if the list of key words needs to be augmented to
support something specail you are doing.