[SciPy-dev] a modest proposal for technology previews

[Jarrod Millman]

On Mon, Nov 3, 2008 at 8:43 PM, Travis E. Oliphant
<oliphant at enthought.com> wrote:
> Right now, though,  I don't see how scipy.preview is preferrable to
> another staging ground like, say, scikits.forscipy.  In fact, I see how
> it might be a bad thing as it basically brings back the sandbox under a
> different name (although one could argue it's a sandbox that actually
> gets distributed).

I absolutely agree that bringing back the sandbox would be a very bad
idea.  But I don't think what I am trying to propose is in any
significant way similar to the sandbox.  The sandbox had no vetting,
no discussion, no plan for inclusion in scipy, and not actively being
developed.  I was imaging that scipy.preview would be reserved for
code that there was a very high bar for inclusion.  Code that had most
likely either had:
 1. some peer review in the case of new call signatures for existing code.
 2. some existence as an external project.

I was trying to address a specific concern I have about how to handle
some of the code that has made it, once again into the trunk right as
we are trying to make a new stable release.  I don't see this
situation going away and I don't see it getting any better.  Rather
than just stating that we will have a very formal vetting process or
we will follow typical release procedures with a feature freeze and
regular releases, I was thinking that there may be another alternative
that would better suit the developer community that we seem to have.

Since we have agreed to include scipy.spatial (this is only an example
we could use scipy.stats.models or numpy.ma or ...), I was wondering
if there was a way to include this code (as opposed to random things
that were thrown into scipy.sandbox) in a way that would indicate that
the developers have agreed:
 1. to include the code in scipy
 2. that the code meets some given standards of test and documentation coverage
 3. whatever else we think should be required....

Currently, the problem is that someone proposes or writes something
that we agree should be part of scipy.  We discuss on the list perhaps
where the code should go or what it should be called.  We may discuss
the API.  Then the code is included in the trunk and we make a
release.  For a large number of scipy user, this may be the first time
they have been made aware of the new code and its API.  They may find
that the code doesn't meet their needs.  Also even, though, the code
has been discussed on the list, most of us our so busy we don't have
very much time to closely look at the code.  So when we agree to
include it, we only have a general sense that the code has a
reasonable API.

I was proposing scipy.preview to try an address these and other
concerns.  It seems doubtful to me that scikits addresses this
problem.  And I don't think that we currently have enough resources to
make scikits better serve this purpose.  I don't have time to work on
scikits and I don't pay much attention to them unfortunately.  If
someone was to step forward and start pushing scikits that would be
great and I would be happy to see that happen.  But even if that was
to happen, we would still need a process to incorporate a scikit into
scipy if we decided we wanted to.  My proposal is aimed at solving
that problem.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/