[SciPy-dev] Some concerns on Scipy development

[pearu at scipy.org]

Hi Eric and Travis.

On Tue, 26 Mar 2002, eric wrote:

> I've have thought about this a little lately also.  There is a philosophical
> difference to packaging among the scientific developers.  Some wish for small
> single purpose and stand alone packages that are installed one by one.  Others
> wish for a single "standard library" of scientific tools that, once installed,
> is a one stop shop for a large number of scientific algorithms.  There are
> benefits to both.  However, I come squarely down in the second camp.  A
> monolithic package is easier to install for end users, and it solves
> compatibility issues (such as SciPy changing the behavior of Numeric in some
> places).  I believe the existence of such a package is required before there can
> be a mass conversion of engineers and scientist to Python as their tool of
> choice for daily tasks.  This is the goal of SciPy.

I have been in peace with this goal of SciPy for a long time. In my
concers I was not trying to propose to change this general goal in any
way. Instead, I was concerned on the internal structure of SciPy and to
see if we could ease the SciPy development and make it more robust for the
future.

One efficient way to achive that would be to require that internal modules
in SciPy would be as independent as possible. A good measure for this
independence is that a particular module can be installed as a standalone. 

Note that I am not proposing this because I would like to use these
modules as standalone modules myself (or any other party), but only to
strengthen SciPy by making it more robust internally.

By doing this, it does not mean that the main goal of SciPy is  
somehow threatened, it will be still a monolithic package for end-users.   
Just its internal structure will be modular and less sensitive to adding
new modules or reviewing some if needed in future.

Now about the question whether SciPy parts can be completely independent?
I think this can be never achived in principle nor it is desired,
but it is a good ideal to follow *whenever* it is possible (and not
just a nice thing to do as you say) and, indeed,  can be practical for
other projects, and all that for the sake of SciPy own success.

<snip>

> So, I think this is a worthy goal for *some* of the modules (notably the ones
> people are discussing such as integrate, linalg, etc), with one caveat. These
> modules need access to some functions provided by scipy and will need to import
> at least one extra module.  Scanning linalg, the needed functions are amax,
> amin, triu, etc. and a handful of functions subsumed from Numeric as well as
> some constants from scipy.limits.  I consider it a bad idea to replicate these
> functions across multiple modules because of the maintenance issues associated
> with duplicate code.  I don't want to go down that path.

Me neither. However your statement that these modules necessarily need
access to scipy functions, is a bit exaggerated. 
In general, there are several ways how the same functionality can be
implemented, and it is my experience that linalg2 can be implemented
without the scipy dependence and that also without replicating any
code. In fact, using high-level scipy convinience functions in linalg2
that is supposed to provide highly efficient and yet to be user-friendly
(yes, both goals can achived at the same time!) algorithms, is not good
because scipy functions just are inefficient due to their general
purpose feature and the initial wins in performance are lost.

Therefore low level modules like linalg, integrate, etc must be carefully
implemented even if it takes more time and seemingly direct Python hooks
could be applied.

> So this is what the site-packages view of scipy would be:
> 
>     site-packages
>         scipy_distutils
>         scipy_test
>         scipy_level0
>             subsumes and customizes Numeric
>             handy.py
>             misc.py
>             scimath.py
>             Matrix.py (?)
>             fastumath.so (pyd)
>             etc.
>         scipy
>             subsume scipy_base
>             everything else

This looks like a positive plan to me.

Any other candidates for naming scipy_level0? It reflects too much
the internals of SciPy but will contain very useful general purpose
functions, I assume, to be useful more widely. 
How about scipy_base?

Another idea would be then to move scipy_test inside scipy_base (and
dropping its scipy_ prefix). Since scipy_base would be mostly pure Python,
it should be feasible.

(Later, be not surprised if I will question the naming of handy.py and
misc.py, but I am not ready for that yet ...;-)

> In regards to higher level modules that use fft, svd, and other complex
> algorithms, they are just gonna have to import scipy.

+2

> This requires some discussion before we make the change.  It's also gonna
> require
> someone to step up and implement the change -- though it probably isn't a major
> effort.

It may be a good idea to release 0.2 before such a change. If it works out
nicely, then 0.3 could follow quickly.

Regards,
	Pearu