[SciPy-dev] Scipy Tutorial (and updating it)

[jh at physics.ucf.edu]

Here are the issues before us as I see them:

You would like:
  a document about each submodule in the SciPy package
    written by the developers/maintainers of those submodules, as experts
  to collect those together into one document
  to name that document "The SciPy User Manual"
  to work on this now

I would like:
  a document for the entire stack
    aimed at new and intermediate users
    using all components of the stack as needed in each chapter
    led by an editing team
    written to an agreed-upon style
    chapters to cover common areas of use
    chapters written by individuals or small teams
    chapter teams selected by an informal proposal process
    to name that document "The SciPy User Manual"
  not to start writing such documents until we have complete docstrings

I think both documents are valuable, as do you.  There are only two
differences:
  Who gets to call it "The SciPy User Manual"?
  When to do what?

On the latter point, I'll compare our situation to the US National
Debt.  We have a documentation deficit.  OVER THREE THOUSAND FUNCTIONS
in NumPy and SciPy DO NOT HAVE GOOD DOCSTRINGS (Gael, I know you know
this figure, I'm shouting for the community to wake up).  We pay
interest on that debt whenever we have to dig into source code or ask
on the mailing list to figure out what a function does.  Writing
higher-level manuals is misspending our resources until we have paid
our documentation debt.  We must get the backlog of documentation
cleared if we are ever to be considered a credible computing
environment.  There is a box of very cool T-shirts in my lab, waiting
to go out to whoever wants to write 1000 words!  (Who else gets to
wear frogs jumping off of sinusoids on a black background?)

Two points on naming.

1. "SciPy" is the only collective name we have, and it's how outsiders
refer to the software.  It's the name of the portal website, and we
considered this implication when the new site went up.  The software
engineering details of how we manage the stack should not push us into
poor choices about how we present the stack to the world.  "SciPy" is
the stack, and a package in the stack.  Not my decision, but we have
to live with it.  It is how outsiders see us, now.

2. "User Guide" has a specific meaning that a collection of submodule
manuals cannot serve.  It literally means an integrative document that
Guides the User in tasks of interest.  Users look for a document
called "User Guide" to learn how to use a suite of packages.  While
some chapters will focus on a specific module or set of modules,
others will not, and not all modules will get a chapter.

The document you describe is not integrative and omits critical things
like reading scientific data formats, plotting, 3D, and so forth that
are not in the SciPy package.  Much of it will also be written at the
expert level, etc.

So, I propose the following:

1. Reserve the name "The SciPy User Guide" for the document I describe
(whether or not I am involved in creating it).

2. Call what you describe something like "The SciPy Package Manual"
that clearly distinguishes it as just being about the package SciPy
and doesn't say "User Guide".

3. Close the SciPy docstring wiki, at least to non-maintainers.

4. Guide anyone who is not a maintainer of SciPy code to contribute
effort to the NumPy docstrings until they are done, then the SciPy
docstrings.

5. Encourage ALL developers and maintainers to write the docstrings
for their packages (in SVN for SciPy), before they do more than fix
bugs in the existing code.  Let's at least get the full function
signatures for ufuncs and the parameter and keyword descriptions done.
Others can fill in the notes and examples based on that.  If they then
want to go write a manual for their packages, great.

Of course, people get to make their own decisions about how to
volunteer their time.

--jh--

Date: Sat, 6 Dec 2008 11:31:39 +0100
From: Gael Varoquaux <gael.varoquaux at normalesup.org>
To: jh at physics.ucf.edu
Cc: scipy-dev at scipy.org
Subject: Re:  [SciPy-dev] Scipy Tutorial (and updating it)
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <wl8fxl2qyqh.fsf at glup.physics.ucf.edu>

On Fri, Dec 05, 2008 at 01:10:46PM -0500, jh at physics.ucf.edu wrote:
> Gael, what you describe below is just a collection of documents about
> each of scipy's modules, written by the maintainers of those modules.
> That is valuable to have, but it is not a user guide.  If you look at
> the User Guide for any of our competing environments (IDL, Matlab,
> Mathematica, etc.), or indeed for any large software package, you'll
> find a document that is integrated across the areas of functionality.

Correct, and I agree that we need this. But this is not the SciPy user
guide. SciPy is only one of the packages. Unlike the other environments,
we have a separation of functionnality. From a software engineering point
of view, this is a good thing (eg it makes shipping pure numerical
algorithms easier, or it makes it possible to run them on embedded
systems). I agree that for a large classe of users, this separation makes
no sens. Thus I agree that an overall document is crucial. But the SciPy
User Guide isn't that.

Gaël