[SciPy-dev] toward scipy 1.0

[Rob Clewley]

On Tue, Nov 4, 2008 at 7:20 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> All,
> I'm not really involved in the development of scipy, so I don't expect my
> opinion to matter much.

I think the point is that it should matter more!

> Breaking scipy into smaller packages sounds a lot like scikits, but is it that
> bad a thing ? Would it make development more difficult ? Would it make
> installation and maintenance more complex ? As long as there's one standard
> for setup.py, things should go OK, shouldn't they ? Because they'd be part of
> scipy and not a scikit, the different modules would have a varnish of
> stability that the scikits don't necessarily have, and it might simplify the
> transformation of a scikit to a scipy module.

Breaking it up might be a bad thing if it means the packages become
too autonomous and disconnected. If the current trend continues and
scipy degenerates into a collection of disjoint subpackages that are
all managed independently, then what's the point of scipy at all?
Doesn't it become a package in name only? I'm not in favor of that at
all. It's no better than having the existing python.org web site
listing of all sorts of different scientific packages that solve
different types of problem and letting people download them all
separately. I thought the point was that scipy should be a coherent
set of core libraries for scientific computation, even if there are
modular and optional add-ons. So I'm all for modularity, but I think
only a smart top-down re-organization will actually help you keep the
core development properly separated from more sideline issues.

> As you'd have guessed, I'm all in favor of a kind of central repository like
cran or ctan. Each scikit could come with a few keywords (provided by their
developer) to simplify the cataloguing, and with a central page it shouldn't
be that difficult to know what is being developed and at what pace, or even
just what is available.

I don't think this idea would follow through to the situation that we
would both like to see.  Your description sounds like an idealization
where we'd effectively start over, with authors submitting brand new,
well thought out packages with keywords that help each other avoid
duplication in the future. But people already have their existing
packages (e.g. current scikits) that they'd immediately submit to the
repository, and there is no mechanism to ensure that their initial
submissions will work together coherently. And that's where the
problem is. So even with keywords or something similar, the
duplication and confusion will have already happened.

To facilitate this process, another idea is to look at the model of
online journals/encyclopedias (e.g. wikipedia, but I'm thinking more
like scholarpedia). We could agree on "chief coordinators" to manage a
broad sub-package, with subs to coordinate at the finer grain level.
This has to be done in a way that doesn't place a huge burden on these
coordinators - they shouldn't be the poor mugs who end up actually
*doing* everything just because they signed up to help.

-Rob