[SciPy-Dev] Growth and sustainability of Scipy development

[Ralf Gommers]

On Sat, Apr 11, 2015 at 2:09 PM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

>
>
> On Thu, Apr 9, 2015 at 8:46 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>
>> Hi,
>>
>> On Sun, Apr 5, 2015 at 3:31 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > Here's a long story about something I've been thinking about for a
>> while.
>> > Some individual parts have been discussed with a few people, but the
>> > complete story is only how I look at things at the moment. I make some
>> > concrete proposals that could be agreed on in this thread if they turn
>> out
>> > to be uncontroversial; we can start separate threads as needed though.
>> >
>> > Summary
>> > --------------
>> >
>> > *Problem Statement*
>> > The growth of core developer and project admin capacity, while pretty
>> strong
>> > over the past few years, is not quite keeping up with the growth in
>> > contributions we're seeing.
>> >
>> > *Solution*
>> > Identify this as an important issue and start addressing it more
>> actively.
>> >
>> > *Main Proposals*
>> > - recruitment: a "become a core dev" mentoring program
>> > - docs:
>> >     - write "how to become a core developer" documentation
>> >     - improve the developer guide further
>> >     - finally merge the Scipy 1.0 roadmap
>> > - communication: set up a private core dev mailing list
>> > - funding:
>> >     - we got some donations (excellent, very useful)
>> >     - have an agreed upon strategy (written down) of what to do with
>> funds
>> >     - then aim at getting more donations and/or getting grants
>> > - governance: we have very little formalized, add something lightweight
>> that
>> > covers most bases
>> >
>> >
>> > Long Version
>> > -------------------
>> >
>> > *Problem Statement*
>> > The number of open PRs is very slowly increasing, and stands now at
>> ~110.
>> > Some of those PRs are stalled, but many simply need reviewing. The
>> number of
>> > active core developers who review and merge PRs is also increasing, but
>> > evidently not at the rate that is necessary for PRs to be reviewed
>> quickly.
>> > The sometimes long delays in getting a PR merged have likely also
>> > discouraged further contributions from some new contributors.
>> >
>> > Note that we did have a lot of growth in active core developers over the
>> > past
>> > few years - I remember Fernando Perez pointing out to me at EuroSciPy'11
>> > that the
>> > Scipy bus factor was exactly 2, now it's more like 8.
>> >
>> > *Recruitment*
>> > Therefore I'd like to look for new core developers in a more pro-active
>> way.
>> > That starts by very clearly saying:
>> >
>> >   We need your help!
>> >
>> > To be more precise, we need both domain specialists that help review
>> PRs for
>> > a single module and core developers that have a broad understanding of
>> how
>> > things are done in Scipy and that take responsibility for signing off
>> on the
>> > quality of PRs and are merging them independently.
>> >
>> > A "core dev mentoring program".  This can go quite quickly for people
>> that
>> > are experienced Python programmers and are familiar with the Scipy
>> > community; it may be as little as a few email conversations or Skype
>> calls
>> > on how to get started.  On the other end of the scale, I'd also like to
>> help
>> > people who are new to the Scipy community or even open source
>> development
>> > get started - as long as they have the right level of
>> coding/communication
>> > skills of course. Then the road to commit rights will of course be
>> longer,
>> > but as long as the ambition to get there is present then it's worth the
>> > effort (I know I certainly wasn't very experienced in contributing to
>> open
>> > source projects at the time I became Scipy release manager, and could
>> > definitely have used some help).
>> >
>> > *Docs*
>> > What developer docs are missing or need significant improvements? Let's
>> add
>> > those (I volunteer for writing duty). We have a reasonable set (links
>> at end
>> > of email), but it's quite scattered. And what we don't have is "how to
>> > become a core developer" documentation, let's add that. Plus finally
>> merge
>> > the 1.0 roadmap: https://github.com/scipy/scipy/pull/2908.
>> >
>> > *Communication*
>> > I'd like to set up a new core dev closed mailing list. Not with the
>> purpose
>> > to do less in public, but simply for organizing the communication that
>> we
>> > already have off-list in a very ad-hoc manner now. Typical topics:
>> deciding
>> > on commit rights, discussions on ranking GSoC students, potential
>> funding
>> > sources and what to do with such funding. I'm not yet sure what to
>> propose
>> > for people with access to that list. We currently have 20 people with
>> commit
>> > rights, of which about half are actually active. Currently decisions are
>> > mostly made between the active devs.
>> >
>> > *Funding*
>> > I'm starting to see more opportunities to use funding in effective ways
>> to
>> > sustain or accelerate Scipy development. Main ones are organizing
>> developer
>> > meetups / sprints, and targeted funding to talented individuals
>> (students
>> > mainly) and for work on particularly hairy bottlenecks (example: Windows
>> > test&build). We have a donation page on scipy.org and have received a
>> number
>> > of donations, but have not yet actively solicited donations because of
>> (a)
>> > no fiscal sponsorship agreement with NumFOCUS signed yet, and (b) while
>> we
>> > can use the money now, I'm uncomfortable to spend large amounts without
>> some
>> > written down strategy on how and why we spend funds.
>> >
>> > *Governance*
>> > Scipy activity is currently very much centered around code, and other
>> > project aspects get relatively little attention. Big email debates about
>> > governance are often not very productive (more high-bandwidth
>> communication
>> > is necessary here), but it would be helpful to keep making steps in this
>> > area.  Documenting processes, how decisions are
>> > made, who decides on things like commit rights, resolving API
>> discussions,
>> > etc. is important.  That goes hand in hand with the developer docs
>> mentioned
>> > above.  The HACKING and MAINTAINERS documents together with the Numpy
>> > developer guide (which also applies to Scipy) do answer quite a few of
>> those
>> > questions, but they're not exactly comprehensive.
>> >
>> > I think we can liberally borrow from other projects on this front, and
>> draft
>> > something reasonable that covers at least basic aspects of how
>> decisions are
>> > made and about things like licensing, (non-)ownership of trademarks and
>> > domain names. This will likely be much easier to reach consensus on than
>> > starting from scratch.
>> > Request: I'd like to keep detailed discussion on the content of such a
>> > governance doc draft for later, and here just discuss/decide whether or
>> not
>> > we want to take this on now and who wants to be involved in putting the
>> > first draft together.
>>
>> Maybe a developer hangout would be the best way forward for all of these?
>>
>
> Thanks Matthew, that's a good idea.
>

In addition, we could briefly discuss the next release (schedule, manager).


> Probably a Google hangout at some time that's evening in Europe and
> daytime in the US would work well. Here's a poll, please fill it out if you
> want to participate: http://doodle.com/ibqnemrvuhkvexm2
>
> Would be good to make an effort to keep the discussion as public as
> possible. So let's make sure we write a summary of what gets discussed and
> post it to this list. And I propose that people who are regular
> contributors but happen to not have a commit bit are also very welcome to
> join the hangout.
>
>
>
>> The discussions on governance in the past have been so miserable and
>> tendentious that it might be better to separate that into a separate
>> track.
>>
>
> I think for Scipy previous discussions (there weren't many) were fine, but
> yes I agree that for Numpy they weren't always pleasant. Let's discuss on
> the hangout what the best approach is.
>
> Ralf
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20150411/b906a331/attachment.html>