[SciPy-dev] Some concerns on Scipy development

[Travis Oliphant]

On Tue, 26 Mar 2002, Pearu Peterson wrote:

>
> Hi,
>
> I have some concerns about how different Scipy modules are dependent on
> each other by a global scipy namespace, though there is often little need
> for this dependency. Namely, most of the current modules import scipy and
> as result one can say that effectively all modules import each other. Such
> a policy is not good because of the following reasons:

>
> 1) It is generally a good style of programming if a module imports only
> thouse resources that it actually uses. This helps, for example, in
> localizing and fixing bugs in a particular module level, which will be
> much easier when doing it in a global scipy level.
> Testing routines should be particullary careful not importing
> irrelevant modules, escpecially, if these are also under testing.
>

The point with SciPy is to bring together a lot of different modules in
one umbrella to make it easier to use.

My vision is that SciPy provides a basic set of functionality that lives
under the SciPy namespace (this functionality encompasses at least
Numeric and MLab and other basic functions).

Modules (e.g. scipy.module_name) could in principle be distributed
separately, but most likely will be written to depend on basic (and
extended) scipy  functionality.

I think we can still re-factor, of course, but there is a reason and
purpose for the design as it exists.



> For a particular example, many modules import MLab or scipy just to
> access only the resources in Numeric. I would remove all such imports in
> favour of importing Numeric directly.

But, I see SciPy subsuming Numeric's function calls.  In other words, a
person shouldn't have to remember whether a certain expression they use
regularly comes (originally) from Numeric or SciPy.

No package in SciPy should use RandomArray, or LinearAlgebra, or FFT
(ideally) as these are all supported in SciPy.

>
> 2) There has been wished (see numpy-discussion list) that scipy wouldn't
> be such a big monolitic environment. Instead, it should be possible to
> install Scipy subpackages as standalone packages if that is possible in
> principle. It has an advantage that other projects could use parts of
> Scipy codes without actually building the whole scipy.

It is hoped that this will be possible.  That is the point of "levels."

The basic package will likely have to be installed in order to get other
modules.  I just don't think it's worth the effort to satisfy one
dissonant user who will probably never like SciPy no matter what we do.

> > Or is this the Scipy policy to force installing/using either the whole
> scipy package or nothing from it? I think that then some experts will not
> use scipy at all (also I find it less stimulating to contribute to
> scipy).

No, but, the user should have to install some "base package."

>
> 3) The idea of different levels in SciPy is not working if the lowest
> level modules import everything. It would be less confusing to drop the
> idea of leveling from start -- I am not proposing to acctually go for it.
> Instead, as I see it, Scipy should be rather thin interface between an
> user and modules that it provides, with some additional (high
> level) convinience functions (that may import "everything"). Lower level
> modules should not depend each other unless there is a good reason for
> that.

First, lower levels don't depend on functions in higher levels, as far as
I know (If they did the code wouldn't import), so I'm not sure where that
statement comes from.

I'm not really sure how your overall idea is different from what we are
doing,  other than the fact that I think Numeric is subsumed by SciPy
(there are  future compatibility issues for this choise as well with
Numeric in a state of flux).

The level idea is just a loose association to try and capture
different dependencies.  It solves the problem of importing things in
the right order in the __init__ file.  It also helps to see which packages
can be "installed separately."


>
> I wonder how other Scipy developers feel about these issues?
> Do you think that my concerns are somehow relevant or not?
> Is it acceptable trying gradually to turn lower level modules to be
> independent from each other if possible?

If you have particular suggestions please state them.  I'm always willing
to consider re-factoring code.

-Travis