[SciPy-dev] Package organization

[Robert Kern]

Pearu Peterson wrote:
> 
> On Thu, 13 Oct 2005, Robert Kern wrote:

>>On top of that, eggs can handle namespace packages, but only one level
>>down, so to speak. I can have several eggs exposing the scipy.*
>>namespace and have them all treated as a single package, but I can't
>>have several eggs exposing the scipy.lib.* namespace and have them all
>>treated uniformly. Adding that functionality to the egg runtime would
>>really complicate matters.
> 
> Ah, ok, it's then all about the limitations of eggs. But what about
> 
>   scipy_lib_sundials-<mumble>.egg/
>   scipy_lib_sundials-<mumble>.egg/scipy/
>   scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/
>   scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/__init__.py
>   scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/_sundials.so
>   scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/sundials.py
> 
> ? (I have never used eggs myself, so, ignore my ignorance on this matter)

I didn't think so, but by plowing through the distutils-SIG mails and
some experimentation, I've found that this will in fact work. So the
only technical objection is null.

>>Perhaps you could explain to me how we benefit putting all of the raw
>>wrappers into scipy.lib.*. I don't really see any.
> 
> The scipy namespace will be cleaner, all packages that scipy will contain, 
> are more or less in the same level. Things that are in scipy.lib would be
> raw wrappers with pythonic interfaces to various libraries that packages
> in scipy level can use. Packages in scipy.lib would depend only on 
> scipy_core, not on each other.

I would only point out that most of the time, it's those Python
interfaces (what would go into sundials.py) that we want to expose to
the user. When SUNDIALS gets wrapped by f2py, we then write a bit of
Python to make the interface convenient and robust. That's pretty much
it. The functions we write in that layer are what we want users to be
using. I don't want that functionality hidden away in
scipy.lib.sundials; I want it in scipy.sundials.

>>If I were starting scipy all over again, I would also make scipy.ode for 
>>ODEPACK and VODE and leave scipy.integrate for QUADPACK and the other 
>>"integrals of functions" routines. I'm not sure it's worth doing the 
>>latter at this point.
> 
> I think in scipy 0.1 or so we had all packages, both wrapper packages as 
> well as toolkits kind of packages, in scipy namespace. I think it was a 
> mess. Moving wrapper packages to scipy.lib would reduce this mess a bit.

That's where we differ, I think. I don't see the distinction between
wrapper packages and toolkits. Most of the time, the wrapper is the
toolkit. It's just that we tended to throw together a bunch of wrappers
into one module and called it a toolkit. If we put all of the wrappers
into scipy.lib.*, we're going to end up with several scipy.* packages
that are just aliases to the scipy.lib.* packages.

> I think that odepack/vode as well as quadpack should be separated 
> from scipy.integrate. My orginal idea was to move the wrappers to 
> scipy.lib and scipy.integrate would just use them from there.
> The main advantage is that when some other scipy package might use, say, 
> odepack, but not quadpack, then does not need to install the whole 
> scipy.integrate. With this separation, the scipy will be more modular.

We're shooting for the same goal. I just think we should split them into
scipy.ode (ODEPACK and VODE) and scipy.integrate (QUADPACK and
quadrature.py) and be done with it. You get the same level of modularity.

Splitting the wrappers into separate scipy.lib.* modules doesn't assist
modularity unless the scipy.* packages that depend on them are also
appropriately split up as well. At that point, most of our scipy.*
packages are just aliases to scipy.lib.* packages.

But like I said, my technical objection is now null, so I'm mostly
arguing from aesthetics. de gustibus non est disputandum, and all that.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter