[SciPy-dev] Package organization

[Pearu Peterson]

On Thu, 13 Oct 2005, Robert Kern wrote:

> Pearu Peterson wrote:
>>
>> On Thu, 13 Oct 2005, Robert Kern wrote:
>
>>> scipy.lib puts a lot of things which are unrelated together. That makes
>>> packaging harder.
>>
>> Hmm, how does it make packaging harder?
>
> If all of the SUNDIALS-related code is under scipy/sundials, then I can
> build eggs that look like this:
>
>  scipy_sundials-<mumble>.egg/
>  scipy_sundials-<mumble>.egg/scipy/
>  scipy_sundials-<mumble>.egg/scipy/sundials/
>  scipy_sundials-<mumble>.egg/scipy/sundials/__init__.py
>  scipy_sundials-<mumble>.egg/scipy/sundials/_sundials.so
>  scipy_sundials-<mumble>.egg/scipy/sundials/sundials.py
>
>  scipy_integrate-<mumble>.egg/
>  scipy_integrate-<mumble>.egg/scipy/
>  scipy_integrate-<mumble>.egg/scipy/integrate/
>  scipy_integrate-<mumble>.egg/scipy/integrate/__init__.py
>  scipy_integrate-<mumble>.egg/scipy/integrate/_quadpack.so
>  scipy_integrate-<mumble>.egg/scipy/integrate/quadpack.py
>  scipy_integrate-<mumble>.egg/scipy/integrate/quadrature.py
>
> rather than
>
>  scipy_sundials-<mumble>.egg/
>  scipy_sundials-<mumble>.egg/scipy/
>  scipy_sundials-<mumble>.egg/scipy/lib/
>  scipy_sundials-<mumble>.egg/scipy/lib/sundials/
>  scipy_sundials-<mumble>.egg/scipy/lib/sundials/_sundials.so
>  scipy_sundials-<mumble>.egg/scipy/integrate/sundials.py
>
>  scipy_integrate-<mumble>.egg/
>  scipy_integrate-<mumble>.egg/scipy/
>  scipy_integrate-<mumble>.egg/scipy/lib/
>  scipy_integrate-<mumble>.egg/scipy/lib/quadpack/
>  scipy_integrate-<mumble>.egg/scipy/lib/quadpack/_quadpack.so
>  scipy_integrate-<mumble>.egg/scipy/integrate/
>  scipy_integrate-<mumble>.egg/scipy/integrate/__init__.py
>  scipy_integrate-<mumble>.egg/scipy/integrate/quadpack.py
>  scipy_integrate-<mumble>.egg/scipy/integrate/quadrature.py

Sure, it's a bad packaging. But why sundials.py should be under integrate? 
That is, wouldn't

  scipy_sundials-<mumble>.egg/
  scipy_sundials-<mumble>.egg/scipy/
  scipy_sundials-<mumble>.egg/scipy/lib/sundials/
  scipy_sundials-<mumble>.egg/scipy/lib/sundials/__init__.py
  scipy_sundials-<mumble>.egg/scipy/lib/sundials/_sundials.so
  scipy_sundials-<mumble>.egg/scipy/lib/sundials/sundials.py

  scipy_integrate-<mumble>.egg/
  scipy_integrate-<mumble>.egg/scipy/
  scipy_integrate-<mumble>.egg/scipy/integrate/
  scipy_integrate-<mumble>.egg/scipy/integrate/__init__.py
  scipy_integrate-<mumble>.egg/scipy/integrate/_quadpack.so
  scipy_integrate-<mumble>.egg/scipy/integrate/quadpack.py
  scipy_integrate-<mumble>.egg/scipy/integrate/quadrature.py

be equivalent to what desired?

> On top of that, eggs can handle namespace packages, but only one level
> down, so to speak. I can have several eggs exposing the scipy.*
> namespace and have them all treated as a single package, but I can't
> have several eggs exposing the scipy.lib.* namespace and have them all
> treated uniformly. Adding that functionality to the egg runtime would
> really complicate matters.

Ah, ok, it's then all about the limitations of eggs. But what about

  scipy_lib_sundials-<mumble>.egg/
  scipy_lib_sundials-<mumble>.egg/scipy/
  scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/
  scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/__init__.py
  scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/_sundials.so
  scipy_lib_sundials-<mumble>.egg/scipy/lib/sundials/sundials.py

? (I have never used eggs myself, so, ignore my ignorance on this matter)

> Perhaps you could explain to me how we benefit putting all of the raw
> wrappers into scipy.lib.*. I don't really see any.

The scipy namespace will be cleaner, all packages that scipy will contain, 
are more or less in the same level. Things that are in scipy.lib would be
raw wrappers with pythonic interfaces to various libraries that packages
in scipy level can use. Packages in scipy.lib would depend only on 
scipy_core, not on each other.

>>> On top of that, you're still importing scipy.lib.sundials,
>>> scipy.lib.odepack, and scipy.lib.quadpack from scipy.integrate. That
>>> means you can't really install that functionality separately.
>>
>> May be I am not quite following you here but I don't understand the
>> "can't" part. Could give an example how would you organize
>> sundials,odepack,quadpack for scipy.integrate?
>
> I would make scipy.sundials completely separate from scipy.integrate.

That's what I would also expect and would make myself.

> If I were starting scipy all over again, I would also make scipy.ode for 
> ODEPACK and VODE and leave scipy.integrate for QUADPACK and the other 
> "integrals of functions" routines. I'm not sure it's worth doing the 
> latter at this point.

I think in scipy 0.1 or so we had all packages, both wrapper packages as 
well as toolkits kind of packages, in scipy namespace. I think it was a 
mess. Moving wrapper packages to scipy.lib would reduce this mess a bit.

I think that odepack/vode as well as quadpack should be separated 
from scipy.integrate. My orginal idea was to move the wrappers to 
scipy.lib and scipy.integrate would just use them from there.
The main advantage is that when some other scipy package might use, say, 
odepack, but not quadpack, then does not need to install the whole 
scipy.integrate. With this separation, the scipy will be more modular.

Pearu