[SciPy-user] We need a system for scipy modularity

[Fernando Perez]

On 7/26/06, Robert Kern <robert.kern at gmail.com> wrote:
> Fernando Perez wrote:

> The mechanism I'm proposing isn't a way to simply "turn off" the subpackages
> that you don't want. It's a way to break up scipy into separately installable
> subpackages. If you want to write a lightweight program that really only needs
> scipy.nd_image, you could simply require "scipy.nd_image". If you want to make
> sure that all of scipy is available, by all means require "scipy".

Would this do dependency analysis?  Declaring that you require
scipy.nd_image would pull in the parts of scipy that nd_image needs
and only those?  I'm just curious at this point.

> The thing is, I don't understand is why scikits is significantly different from
> scipy, here. The only thing different is that scipy.__init__ isn't empty. That's
> okay; that's easy to deal with.

To me it's just an organizational issue: making scipy a monolithic,
but not too big core which is guaranteed to have a certain list of
things always in it, and moving all the ancillary stuff out into the
toolkits.  I also think that would help with maintenance, build times,
etc.  Basically I view scipy as numpy++ (or numpy+fortran, if you
will), and would leave the modularity to toolkits with scikit as a
namespace, so that even third-parties could provide packages whose

setup.py install

goes under site-packages/scikit (yes, issues of --prefix need sorting
out, perhaps the setuptools namespace package concept is the proper
solution; I'm not familiar with it).

> The current problem is largely the building, not the namespace manipulation.
> That's already a solved problem. We don't need to roll our own, here.

I don't think I was proposing rolling anything new on our own, I think
it's more a question of policy than one of implementation.  One
approach is to make scipy itself modular, which I tend not to like a
whole lot.  Another (the one I propose) is to keep scipy monolithic,
but small-ish (like numpy is, but obviously bigger than numpy), and
have all modularity in the notion of toolkits (even if this means
moving some things in today's sandbox out to the toolkits).

There are probably actual building implementation details that apply
to either approach, and in that case your current solution would
probably work for either.  But the policy decision is independent of
the mechanism.

I think I've said enough on this matter, and I won't be writing any
code for this right now.  If you want, I'm willing to work (alone or
with you) on my approach during the sprint days.  If you roll
something else before then, we'll all use it and I'll be OK with
whatever it is.  If people prefer the 'modular scipy' approach, then
that's fine.

Cheers,

f