[SciPy-dev] Package organization

[Robert Kern]

Pearu Peterson wrote:
> 
> On Thu, 13 Oct 2005, Robert Kern wrote:

>>scipy.lib puts a lot of things which are unrelated together. That makes 
>>packaging harder.
> 
> Hmm, how does it make packaging harder?

If all of the SUNDIALS-related code is under scipy/sundials, then I can
build eggs that look like this:

  scipy_sundials-<mumble>.egg/
  scipy_sundials-<mumble>.egg/scipy/
  scipy_sundials-<mumble>.egg/scipy/sundials/
  scipy_sundials-<mumble>.egg/scipy/sundials/__init__.py
  scipy_sundials-<mumble>.egg/scipy/sundials/_sundials.so
  scipy_sundials-<mumble>.egg/scipy/sundials/sundials.py

  scipy_integrate-<mumble>.egg/
  scipy_integrate-<mumble>.egg/scipy/
  scipy_integrate-<mumble>.egg/scipy/integrate/
  scipy_integrate-<mumble>.egg/scipy/integrate/__init__.py
  scipy_integrate-<mumble>.egg/scipy/integrate/_quadpack.so
  scipy_integrate-<mumble>.egg/scipy/integrate/quadpack.py
  scipy_integrate-<mumble>.egg/scipy/integrate/quadrature.py

rather than

  scipy_sundials-<mumble>.egg/
  scipy_sundials-<mumble>.egg/scipy/
  scipy_sundials-<mumble>.egg/scipy/lib/
  scipy_sundials-<mumble>.egg/scipy/lib/sundials/
  scipy_sundials-<mumble>.egg/scipy/lib/sundials/_sundials.so
  scipy_sundials-<mumble>.egg/scipy/integrate/sundials.py

  scipy_integrate-<mumble>.egg/
  scipy_integrate-<mumble>.egg/scipy/
  scipy_integrate-<mumble>.egg/scipy/lib/
  scipy_integrate-<mumble>.egg/scipy/lib/quadpack/
  scipy_integrate-<mumble>.egg/scipy/lib/quadpack/_quadpack.so
  scipy_integrate-<mumble>.egg/scipy/integrate/
  scipy_integrate-<mumble>.egg/scipy/integrate/__init__.py
  scipy_integrate-<mumble>.egg/scipy/integrate/quadpack.py
  scipy_integrate-<mumble>.egg/scipy/integrate/quadrature.py

Now, with the former I have two, well-defined subpackages that never
change. They're either present or they're not. With the latter, the
contents of scipy.integrate change depending on whether or not
scipy_sundials is installed.

On top of that, eggs can handle namespace packages, but only one level
down, so to speak. I can have several eggs exposing the scipy.*
namespace and have them all treated as a single package, but I can't
have several eggs exposing the scipy.lib.* namespace and have them all
treated uniformly. Adding that functionality to the egg runtime would
really complicate matters.

Perhaps you could explain to me how we benefit putting all of the raw
wrappers into scipy.lib.*. I don't really see any.

>>On top of that, you're still importing scipy.lib.sundials, 
>>scipy.lib.odepack, and scipy.lib.quadpack from scipy.integrate. That 
>>means you can't really install that functionality separately.
> 
> May be I am not quite following you here but I don't understand the 
> "can't" part. Could give an example how would you organize 
> sundials,odepack,quadpack for scipy.integrate?

I would make scipy.sundials completely separate from scipy.integrate. If
I were starting scipy all over again, I would also make scipy.ode for
ODEPACK and VODE and leave scipy.integrate for QUADPACK and the other
"integrals of functions" routines. I'm not sure it's worth doing the
latter at this point.

(Oh, and thank you for reducing the default chattiness of ScipyTest. I
appreciate it.)

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter