[SciPy-Dev] Growth and sustainability of Scipy development

[Matthew Brett]

Hi,

On Sat, Apr 11, 2015 at 5:09 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Thu, Apr 9, 2015 at 8:46 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Sun, Apr 5, 2015 at 3:31 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > Here's a long story about something I've been thinking about for a
>> > while.
>> > Some individual parts have been discussed with a few people, but the
>> > complete story is only how I look at things at the moment. I make some
>> > concrete proposals that could be agreed on in this thread if they turn
>> > out
>> > to be uncontroversial; we can start separate threads as needed though.
>> >
>> > Summary
>> > --------------
>> >
>> > *Problem Statement*
>> > The growth of core developer and project admin capacity, while pretty
>> > strong
>> > over the past few years, is not quite keeping up with the growth in
>> > contributions we're seeing.
>> >
>> > *Solution*
>> > Identify this as an important issue and start addressing it more
>> > actively.
>> >
>> > *Main Proposals*
>> > - recruitment: a "become a core dev" mentoring program
>> > - docs:
>> >     - write "how to become a core developer" documentation
>> >     - improve the developer guide further
>> >     - finally merge the Scipy 1.0 roadmap
>> > - communication: set up a private core dev mailing list
>> > - funding:
>> >     - we got some donations (excellent, very useful)
>> >     - have an agreed upon strategy (written down) of what to do with
>> > funds
>> >     - then aim at getting more donations and/or getting grants
>> > - governance: we have very little formalized, add something lightweight
>> > that
>> > covers most bases
>> >
>> >
>> > Long Version
>> > -------------------
>> >
>> > *Problem Statement*
>> > The number of open PRs is very slowly increasing, and stands now at
>> > ~110.
>> > Some of those PRs are stalled, but many simply need reviewing. The
>> > number of
>> > active core developers who review and merge PRs is also increasing, but
>> > evidently not at the rate that is necessary for PRs to be reviewed
>> > quickly.
>> > The sometimes long delays in getting a PR merged have likely also
>> > discouraged further contributions from some new contributors.
>> >
>> > Note that we did have a lot of growth in active core developers over the
>> > past
>> > few years - I remember Fernando Perez pointing out to me at EuroSciPy'11
>> > that the
>> > Scipy bus factor was exactly 2, now it's more like 8.
>> >
>> > *Recruitment*
>> > Therefore I'd like to look for new core developers in a more pro-active
>> > way.
>> > That starts by very clearly saying:
>> >
>> >   We need your help!
>> >
>> > To be more precise, we need both domain specialists that help review PRs
>> > for
>> > a single module and core developers that have a broad understanding of
>> > how
>> > things are done in Scipy and that take responsibility for signing off on
>> > the
>> > quality of PRs and are merging them independently.
>> >
>> > A "core dev mentoring program".  This can go quite quickly for people
>> > that
>> > are experienced Python programmers and are familiar with the Scipy
>> > community; it may be as little as a few email conversations or Skype
>> > calls
>> > on how to get started.  On the other end of the scale, I'd also like to
>> > help
>> > people who are new to the Scipy community or even open source
>> > development
>> > get started - as long as they have the right level of
>> > coding/communication
>> > skills of course. Then the road to commit rights will of course be
>> > longer,
>> > but as long as the ambition to get there is present then it's worth the
>> > effort (I know I certainly wasn't very experienced in contributing to
>> > open
>> > source projects at the time I became Scipy release manager, and could
>> > definitely have used some help).
>> >
>> > *Docs*
>> > What developer docs are missing or need significant improvements? Let's
>> > add
>> > those (I volunteer for writing duty). We have a reasonable set (links at
>> > end
>> > of email), but it's quite scattered. And what we don't have is "how to
>> > become a core developer" documentation, let's add that. Plus finally
>> > merge
>> > the 1.0 roadmap: https://github.com/scipy/scipy/pull/2908.
>> >
>> > *Communication*
>> > I'd like to set up a new core dev closed mailing list. Not with the
>> > purpose
>> > to do less in public, but simply for organizing the communication that
>> > we
>> > already have off-list in a very ad-hoc manner now. Typical topics:
>> > deciding
>> > on commit rights, discussions on ranking GSoC students, potential
>> > funding
>> > sources and what to do with such funding. I'm not yet sure what to
>> > propose
>> > for people with access to that list. We currently have 20 people with
>> > commit
>> > rights, of which about half are actually active. Currently decisions are
>> > mostly made between the active devs.
>> >
>> > *Funding*
>> > I'm starting to see more opportunities to use funding in effective ways
>> > to
>> > sustain or accelerate Scipy development. Main ones are organizing
>> > developer
>> > meetups / sprints, and targeted funding to talented individuals
>> > (students
>> > mainly) and for work on particularly hairy bottlenecks (example: Windows
>> > test&build). We have a donation page on scipy.org and have received a
>> > number
>> > of donations, but have not yet actively solicited donations because of
>> > (a)
>> > no fiscal sponsorship agreement with NumFOCUS signed yet, and (b) while
>> > we
>> > can use the money now, I'm uncomfortable to spend large amounts without
>> > some
>> > written down strategy on how and why we spend funds.
>> >
>> > *Governance*
>> > Scipy activity is currently very much centered around code, and other
>> > project aspects get relatively little attention. Big email debates about
>> > governance are often not very productive (more high-bandwidth
>> > communication
>> > is necessary here), but it would be helpful to keep making steps in this
>> > area.  Documenting processes, how decisions are
>> > made, who decides on things like commit rights, resolving API
>> > discussions,
>> > etc. is important.  That goes hand in hand with the developer docs
>> > mentioned
>> > above.  The HACKING and MAINTAINERS documents together with the Numpy
>> > developer guide (which also applies to Scipy) do answer quite a few of
>> > those
>> > questions, but they're not exactly comprehensive.
>> >
>> > I think we can liberally borrow from other projects on this front, and
>> > draft
>> > something reasonable that covers at least basic aspects of how decisions
>> > are
>> > made and about things like licensing, (non-)ownership of trademarks and
>> > domain names. This will likely be much easier to reach consensus on than
>> > starting from scratch.
>> > Request: I'd like to keep detailed discussion on the content of such a
>> > governance doc draft for later, and here just discuss/decide whether or
>> > not
>> > we want to take this on now and who wants to be involved in putting the
>> > first draft together.
>>
>> Maybe a developer hangout would be the best way forward for all of these?
>
>
> Thanks Matthew, that's a good idea.
>
> Probably a Google hangout at some time that's evening in Europe and daytime
> in the US would work well. Here's a poll, please fill it out if you want to
> participate: http://doodle.com/ibqnemrvuhkvexm2

Done - thanks.

> Would be good to make an effort to keep the discussion as public as
> possible. So let's make sure we write a summary of what gets discussed and
> post it to this list. And I propose that people who are regular contributors
> but happen to not have a commit bit are also very welcome to join the
> hangout.

Thanks - that is a good idea.

>> The discussions on governance in the past have been so miserable and
>> tendentious that it might be better to separate that into a separate
>> track.
>
>
> I think for Scipy previous discussions (there weren't many) were fine, but
> yes I agree that for Numpy they weren't always pleasant. Let's discuss on
> the hangout what the best approach is.

Yes, right, it was the numpy discussions, but there is so much overlap
between the lists, I am sure the atmosphere over there has had an
effect.

Cheers,

Matthew