[SciPy-dev] Package organization

[Fernando Perez]

Robert Kern wrote:

> I'm not entirely convinced that everything needs to live in the scipy
> namespace, though. I think I overstated the build-time benefits of being
> in scipy itself. I think we can make most of those benefits accessible
> to packages that aren't living in scipy.* . I haven't been keeping up
> with the AstroPy discussions, but I doubt they'd really want to do
> 
>   from scipy.kits.astropy import wcs
> 
> etc. "Flat is better than nested," and all that.

Yes, but there's a dark side to that motto, rarely mentioned: 'too flat leads 
to name clashes and silent shadowings'.  That's exactly what happens with 
python: the search space is a single, flat list searched in order (as 
specified in some internal build-time paths extended with $PYTHONPATH, 
resulting in sys.path).  This means that at any time, the appearance of a new 
package with the same name as another in a directory which is listed earlier 
in sys.path can shadow the older package with no warning.

I think that scipy is a large enough collection of tools that it's worth 
creating at least one level of namespace protection for us against the above.

Indeed, it is shorter to type

from astropy import wcs

and perhaps the astropy community (and others) would find the scipy.kits 
prefix annoying, I don't know.  But I do see merit to having a namespace 
umbrella that may justify that price to be paid:

- scipy.kits? or help(scipy.kits) will show you all the toolkits you have 
installed.  I think this can help new users a lot, and even experienced ones, 
as scipy grows.  Think of how much people love the integrated help indices in 
Mathematica or matlab...

- automatic documentation facilities can collect and generate html/pdf docs of 
the scipy core/full AND the existing toolkits, including code examples.

- with the addition of a single environment variable (say SCIPY_KIT_PATH), we 
can even allow users to manage local collections of toolkits outside of the 
sysadmin-controlled filesystem (critical for non-root usage).  Obviously 
PYTHONPATH already provides this for import, but explicitly designating a 
toolkit search path will tell scipy where to look at runtime for toolkits, for 
the purposes of documentation/help/indexing (much like MayaVi allows for 
user-level local extensions for data sources, filters and modules).


If the overall idea seems to anyone to have merit but the issue is with too 
much nesting, would a simple renaming help?

from scikit.astro import wcs

for example.  It does bring the nesting cost down to a single lookup level 
(one dot instead of two, which means one less dictionary lookup) and 7 
characters total.

> I *think* a package can get all the benefits of "being in scipy" simply
> by depending on parts of scipy and using the tools appropriately. But to
> really figure this out, we need to come down to cases. Let's get our
> house in order first, and then we can talk about how we deal with guests.

I just want to make sure that we at least build a guest room in the house, 
instead of asking the guests to sleep in the garage :)  We can paint it later, 
but we should at least plan for its existence now.


Anyway, if people don't like the idea I'll just drop the discussion so that 
those coding can get back to their job, and I can get back to the 
long-abandoned ipython.  I'll live with the occasional PYTHONPATH-induced name 
clash, but I do think it would be worth taking this opportunity now to 
future-proof scipy a little in this regard.

Cheers,

f