[SciPy-Dev] Growth and sustainability of Scipy development

[Ralf Gommers]

Hi,

Here's a long story about something I've been thinking about for a while.
Some individual parts have been discussed with a few people, but the
complete story is only how I look at things at the moment. I make some
concrete proposals that could be agreed on in this thread if they turn out
to be uncontroversial; we can start separate threads as needed though.

Summary
--------------

*Problem Statement*
The growth of core developer and project admin capacity, while pretty
strong over the past few years, is not quite keeping up with the growth in
contributions we're seeing.

*Solution*
Identify this as an important issue and start addressing it more actively.

*Main Proposals*
- recruitment: a "become a core dev" mentoring program
- docs:
    - write "how to become a core developer" documentation
    - improve the developer guide further
    - finally merge the Scipy 1.0 roadmap
- communication: set up a private core dev mailing list
- funding:
    - we got some donations (excellent, very useful)
    - have an agreed upon strategy (written down) of what to do with funds
    - then aim at getting more donations and/or getting grants
- governance: we have very little formalized, add something lightweight
that covers most bases


Long Version
-------------------

*Problem Statement*
The number of open PRs is very slowly increasing, and stands now at ~110.
Some of those PRs are stalled, but many simply need reviewing. The number
of active core developers who review and merge PRs is also increasing, but
evidently not at the rate that is necessary for PRs to be reviewed
quickly.  The sometimes long delays in getting a PR merged have likely also
discouraged further contributions from some new contributors.

Note that we did have a lot of growth in active core developers over the
past
few years - I remember Fernando Perez pointing out to me at EuroSciPy'11
that the
Scipy bus factor was exactly 2, now it's more like 8.

*Recruitment*
Therefore I'd like to look for new core developers in a more pro-active way.
That starts by very clearly saying:

  We need your help!

To be more precise, we need both domain specialists that help review PRs for
a single module and core developers that have a broad understanding of how
things are done in Scipy and that take responsibility for signing off on the
quality of PRs and are merging them independently.

A "core dev mentoring program".  This can go quite quickly for people that
are experienced Python programmers and are familiar with the Scipy
community; it may be as little as a few email conversations or Skype calls
on how to get started.  On the other end of the scale, I'd also like to
help people who are new to the Scipy community or even open source
development get started - as long as they have the right level of
coding/communication skills of course. Then the road to commit rights will
of course be longer, but as long as the ambition to get there is present
then it's worth the effort (I know I certainly wasn't very experienced in
contributing to open source projects at the time I became Scipy release
manager, and could definitely have used some help).

*Docs*
What developer docs are missing or need significant improvements? Let's add
those (I volunteer for writing duty). We have a reasonable set (links at
end of email), but it's quite scattered. And what we don't have is "how to
become a core developer" documentation, let's add that. Plus finally merge
the 1.0 roadmap: https://github.com/scipy/scipy/pull/2908.

*Communication*
I'd like to set up a new core dev closed mailing list. Not with the purpose
to do less in public, but simply for organizing the communication that we
already have off-list in a very ad-hoc manner now. Typical topics: deciding
on commit rights, discussions on ranking GSoC students, potential funding
sources and what to do with such funding. I'm not yet sure what to propose
for people with access to that list. We currently have 20 people with
commit rights, of which about half are actually active. Currently decisions
are mostly made between the active devs.

*Funding*
I'm starting to see more opportunities to use funding in effective ways to
sustain or accelerate Scipy development. Main ones are organizing developer
meetups / sprints, and targeted funding to talented individuals (students
mainly) and for work on particularly hairy bottlenecks (example: Windows
test&build). We have a donation page on scipy.org and have received a
number of donations, but have not yet actively solicited donations because
of (a) no fiscal sponsorship agreement with NumFOCUS signed yet, and (b)
while we can use the money now, I'm uncomfortable to spend large amounts
without some written down strategy on how and why we spend funds.

*Governance*
Scipy activity is currently very much centered around code, and other
project aspects get relatively little attention. Big email debates about
governance are often not very productive (more high-bandwidth communication
is necessary here), but it would be helpful to keep making steps in this
area.  Documenting processes, how decisions are
made, who decides on things like commit rights, resolving API discussions,
etc. is important.  That goes hand in hand with the developer docs mentioned
above.  The HACKING and MAINTAINERS documents together with the Numpy
developer guide (which also applies to Scipy) do answer quite a few of those
questions, but they're not exactly comprehensive.

I think we can liberally borrow from other projects on this front, and
draft something reasonable that covers at least basic aspects of how
decisions are made and about things like licensing, (non-)ownership of
trademarks and domain names. This will likely be much easier to reach
consensus on than starting from scratch.
Request: I'd like to keep detailed discussion on the content of such a
governance doc draft for later, and here just discuss/decide whether or not
we want to take this on now and who wants to be involved in putting the
first draft together.

Ralf


Developer docs:
https://github.com/scipy/scipy/blob/master/HACKING.rst.txt
https://github.com/rgommers/scipy/blob/master/doc/MAINTAINERS.rst.txt
http://docs.scipy.org/doc/numpy-dev/dev/
https://github.com/numpy/numpy/blob/master/doc/source/dev/development_environment.rst
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20150406/82d1816b/attachment.html>