[SciPy-dev] Some concerns on Scipy development

[eric]

Hey Pearu,

I've have thought about this a little lately also.  There is a philosophical
difference to packaging among the scientific developers.  Some wish for small
single purpose and stand alone packages that are installed one by one.  Others
wish for a single "standard library" of scientific tools that, once installed,
is a one stop shop for a large number of scientific algorithms.  There are
benefits to both.  However, I come squarely down in the second camp.  A
monolithic package is easier to install for end users, and it solves
compatibility issues (such as SciPy changing the behavior of Numeric in some
places).  I believe the existence of such a package is required before there can
be a mass conversion of engineers and scientist to Python as their tool of
choice for daily tasks.  This is the goal of SciPy.

That said, the monolithic nature does pose some problems occasionally
complicating development and following the CVS as Jochen has pointed out. Also,
some may want to use some modules (weave, integrate, linalg) outside of scipy.
This is useful in cases where you want to minimize install size or use something
like py2exe (which currently doesn't work with SciPy).  We should facilitate
separating out packages when it is convenient, but not when it requires
duplication of code or a lot of extra work. Perhaps re-organizing the
architecture can make it convenient more often and come closer to making both
camps happy.

I agree that, when possible, it is nice to develop packages independent of
SciPy -- that is how weave was developed.  Later it was folded into SciPy, but
it still runs separately.  The new build structure (with separate setup_xxx.py
files for each sub-package) implemented several months ago was developed, in
part, to facilitate this sort of thing.  Weave was easy in this respect because
it doesn't need many numeric capabilities.

So, I think this is a worthy goal for *some* of the modules (notably the ones
people are discussing such as integrate, linalg, etc), with one caveat. These
modules need access to some functions provided by scipy and will need to import
at least one extra module.  Scanning linalg, the needed functions are amax,
amin, triu, etc. and a handful of functions subsumed from Numeric as well as
some constants from scipy.limits.  I consider it a bad idea to replicate these
functions across multiple modules because of the maintenance issues associated
with duplicate code.  I don't want to go down that path.  However, one thought
is to make the idea of "levels" more explicit.  We could define a package called
"scipy_lite" or "scipy_level0" that would subsume Numeric and add the helper
functions that are often used.  It would not reference other scipy modules.
This package would live in the scipy development tree, but would install as a
separate package.  So scipy_level0 would sit next to scipy in the site-packages
directory.  scipy_level0 would be easy to build without major dependencies --
much like Numeric.  It would hold fastumath and maybe a few other extension
modules, but it would be predominantly python code.  The linalg, integrate, etc
modules would import scipy_level0 instead of scipy.  This way, people only have
to port scipy_level0 instead of the whole of scipy if they want to use integrate
in their package.

I can't imagine much dissent about this approach by the people wanting single
packages.  If your willing to install Numeric and linalg on your own, then you
should be willing to install the scipy_level0 package.

Installing scipy_level0 outside the scipy package has some precedence since
we've already done this with scipy_distutils and scipy_test.  I'd rather leave
it as a sub-directory of scipy, but pulling it out is necessary because of the
way that python handles (or doesn't handle...) imports within packages -- that
is if we want to make it easy to use sub-modules of scipy separately.  So this
is what the site-packages view of scipy would be:

    site-packages
        scipy_distutils
        scipy_test
        scipy_level0
            subsumes and customizes Numeric
            handy.py
            misc.py
            scimath.py
            Matrix.py (?)
            fastumath.so (pyd)
            etc.
        scipy
            subsume scipy_base
            everything else

In regards to higher level modules that use fft, svd, and other complex
algorithms, they are just gonna have to import scipy.

This requires some discussion before we make the change.  It's also gonna
require
someone to step up and implement the change -- though it probably isn't a major
effort.

eric

--
Eric Jones <eric at enthought.com>
Enthought, Inc. [www.enthought.com and www.scipy.org]
(512) 536-1057