[SciPy-Dev] Growth and sustainability of Scipy development
Ralf Gommers
ralf.gommers at gmail.com
Thu Apr 23 02:12:04 EDT 2015
On Wed, Apr 22, 2015 at 6:43 PM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:
>
>
> On Wed, Apr 22, 2015 at 6:33 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>>
>>
>> On Wed, Apr 22, 2015 at 12:50 PM, David Cournapeau <cournape at gmail.com>
>> wrote:
>>
>>> Is there a link we should be aware of for bluejeans ? Just making sure I
>>> have not missed the connection info.
>>>
>>
>> Good question. I didn't send it out yet, wanted to do a bit of testing
>> first (conclusion: should work on WIndows, OS X and 64-bit Linux but not on
>> 32-bit Linux).
>>
>> Here is the link to the meeting: https://bluejeans.com/865079980. That
>> should be all you need (no password), but in case it asks you for a meeting
>> ID that is 865079980.
>>
>> The meeting minutes are at, everyone should be able to edit them. I tried
>> to get the synchronization between StackEdit and Google Drive to work, but
>> that seems to be a bit of a mess. So editing live in Google Docs it is -
>> we'll clean up to publish on the mailing list and as rendered Markdown
>> afterwards.
>>
>
> EDIT: this is the link that should work:
> https://drive.google.com/open?id=1vsLrXIAVGwkHb2hdscuWu9ZXik-3NH80X8Oh0eTy0cM&authuser=0
>
>
>>
>> Reminder about time zones (this always goes wrong):
>> UTC 7-8 pm
>> Local time Western Europe: 9-10 pm
>> CDT: 2-3 pm
>> time from now: 2.5 hours
>>
>> Cheers,
>> Ralf
>>
>>
Hi all, we had an interesting discussion yesterday with about 16 people.
And I personally liked connecting faces to names of people I've been
interacting with recently or for years. Below the minutes we took during
and edited after the meeting.
Cheers,
Ralf
Scipy developer hangout 22 April 2015
================================
**Attendance**: Ralf Gommers, David Cournapeau, Matthew Brett, Evgeni
Burovski, Eric Moore, Josef Perktold, Charles Harris, Nathaniel Smith,
Abraham Escalante, Eric Larson, Jamie Morton, Yoshiki Vazquez-Baeza, Neal
Becker, Ian Henriksen, Ariel Rokem, Julian Taylor
Mailing list thread that triggered this hangout:
http://article.gmane.org/gmane.comp.python.scientific.devel/19607
First thing to do is agree on the agenda for this hangout. Proposed agenda:
1.
Broadening developer base
2.
Developer docs & roadmap
3.
Communication
4.
Funding
5.
Governance
6.
Next release
Agenda OK as proposed, no one has additions.
Broadening developer base
-------------------------------------
> **Proposal**: (made on mailing list) start a "core dev mentoring program".
Discussion started with this proposal, which triggered a broader discussion
of the problem to be solved. Other things to consider:
-
What can we learn from other projects like Scikit-learn and Sympy that
have been more successful in attracting new developers?
-
David C: At least for scikit learn, they have at least one paid, full
time person to work on the project (Olivier Grisel), and my understanding
is that he spends most of his time managing the project (reviewing PR, etc…)
-
Recently Matplotlib development changed a lot. Matthew: mostly caused by
one new lead developer spending a lot of time and energy.
-
Josef: Scipy needs also domain specialists; PRs tend to stall if the one
area that that expert takes care of takes a break.
-
David: related to Josef’s point, do we want to have “sub-maintainers”
for a particular subpackage ? Having a default go-to for advice may be
helpful to newcomers, not sure how it would work in practice
-
Ralf: Scipy covers a very diverse set of topics, it’s almost impossible
to be an expert on all of them. Pauli is pretty much the only one that has
deep knowledge on most submodules.
Discussion then turned to funding and how we could use it (see section
“Funding” below), the actual proposal wasn’t discussed in much detail.
Developer docs & roadmap
-------------------------------------
developer docs not discussed, roadmap was discussed (see under funding)
Communication
---------------------
> **Proposal**: (made on mailing list) set up a closed mailing list for
developers.
Opinions are mixed on this proposal.
Matthew: you don’t want to give the impression that there are significant
discussions going on that should be public.
Ralf: agreed. However there are some topics (GSoC student ranking,
decisions on commit rights, potential funding sources) that anyway have to
happen and cannot be public. So we can make an explicit list of topics that
that mailing list is for.
Matthew: it can still give the wrong impression, ad hoc offline discussions
like there are now is probably preferable.
Josef: if closed list gets archived, people that later get access may not
like what they see then.
Matthew: decisions should always be reasonable and based on evidence.
Ethical question to ask yourself: would you be comfortable with seeing the
discussion on a decision published in the Wall Street Journal?
Abraham: personally I would like to see the reasons of why I was or wasn’t
accepted for GSoC later on, in order to learn from it.
Ralf: there’s enough discussion/resistance here to conclude that we
shouldn’t move forward with this proposal now.
Nathaniel: suggest we tackle the specific needs one at a time: e.g., make a
little mailing list for each GSoC, come up with some less-informal
mechanism for deciding who gets commit rights, etc.
Nathaniel’s suggestion is accepted as the way forward.
Funding
-----------
Matthew: Do we have anything to spend money on?
Ralf: yes, but the first issue is that I wouldn’t be comfortable spending
significant amounts of money without some governance doc/structure that
says something about how decisions about money are made.
Matthew: say we short-circuit that now by saying you are the dictator (not
an actual proposal) and can make those decisions?
Ralf: then, yes.
Chuck: We can’t afford senior developer types.
Ralf: Yeah, but we could potentially hire masters or PhD or postdocs for
any length of time between 3 months and a year or two at a time.
Bigger issue: what do we spend money on? And how do we present that to
possible money sources? Ralf suggests the roadmap (
https://github.com/scipy/scipy/pull/2908/files) as a todo list.
Nathaniel is concerned that this is a big list of little cleanups and stuff
that may be important but don’t look very urgent or make a good story for
funders. (And also secondarily he’s somewhat concerned about the idea of
making “1.0” a conglomerate of everyone’s favorite wishlist item; it seems
like lots of those cleanups on the list could happen after 1.0 without any
real harm? But that’s orthogonal to the governance/funding/etc. issues.)
[Ralf, adding after the meeting] looking at that roadmap again, Nathaniel
has a point. Significant features, like unifying spline representations,
are on there but not identified clearly as major new features. Other
potentially important new features are missing from the roadmap because we
didn’t consider them “necessary for 1.0”. This needs improving.
David C. is wondering about the longer term “vision” of the project: if we
had somebody to work say 6 months, do we want something that goes beyond
bug fixing and “minor” features ? What makes a good “subproject” for being
included in scipy ? How do we position ourselves compared to the popular
scikits ?
Ralf: there are things like B-splines in interpolate, introducing Propack
in scipy.sparse, etc. that would require a talented developer to spend say
3-6 months full time on in order to push that topic forward. Also, projects
like Scikit-learn (sparse) and scikit-image (ndimage) lean heavily on Scipy.
Ian: the Cython API for BLAS/LAPACK is another good example.
Matthew: improving the Windows build situation (MinGW64, allowing use of
Fortran 90) would be valuable.
Nathaniel, adding after the meeting: the biggest downstream impact change I
can think of for scipy is getting a scipy.sparse API that doesn’t use
np.matrix -- this will directly change how a lot of people work on a daily
basis, and probably cause a large increase in the number of projects that
support sparse matrices. (This is blocked on __numpy_ufunc__ support in
numpy, but that will be released soon.)
Governance
----------------
Question: do we need people to look into this in more detail and come up
with some draft proposal as a basis for discussion? Volunteers: Matthew
after June 6th, Ralf.
Chuck: would like time to read the links and book by Karl Fogel. Can we
give everyone time to do this before following up? Decision: agree with
Chuck’s suggestion, follow-up call in one month. Ralf will organize (and
find a volunteer to host the call on bluejeans.com).
Concrete parts of governance:
-
Fiscal responsibility team for formalizing organization, NumFOCUS
relationship
-
Deciding on a mechanism for who gets commit rights and what those mean.
-
Chuck is concerned about commit right list growing to be very long
and full of inactive people, which could eventually become a security
concern. Nathaniel argued that this is not really a big deal in practice
(sabotage against OSS projects is incredibly easy and yet almost
unheard of
in practice). On the other hand, just kicking people out for inactivity
could be a big source of bad feelings and drama. One way to handle both
concerns in a minimal-drama way would be to have a
commit-rights-policy-document that lays out how commit rights
are bestowed,
and also says that as a matter of policy to reduce security concerns, we
will remove commit rights from the github interface from anyone who is
inactive for X months, but this is purely procedural -- the person will
still be considered to be a committer and their permissions will be
restored automatically on request.
-
Some other projects’ rules for committers:
-
Karl Fogel’s (excellent and free) Producing OSS book has a section
discussing exactly this: http://producingoss.com/en/committers.html
-
The Rubinius project uses a very interesting model when your first
pull request is merged, then you automatically get commit
rights at that
time. (Of course you still have to follow other rules,
doesn’t mean you can
just scribble nonsense in master. Or if you do of course you will
immediately get kicked out and your rights reverted.) Like wikis, this
apparently works better than you would expect. Interesting
blog posts that
are not about this exactly but do touch on it [better sources welcome]:
-
http://rubini.us/2014/11/10/rubinius-3-0-part-1-the-rubinius-team/
-
http://rubini.us/2014/11/11/rubinius-3-0-part-2-the-process/
-
IPython’s governance document:
https://github.com/ipython/ipython/wiki/IPEP-29:-Project-Governance
-
Regarding “what commit rights mean”, this is also very important.
Probably something like, “you can, after due diligence/review,
merge other
people’s pull requests”. Does *not* mean that you can commit directly to
master without oversight, and this should be documented somewhere.
-
Do we want to formalize a bit more PR review (e.g. at least N people
agreeing on it) ?
-
We already do this informally based on reviewer judgement (“I
think this is good but I’m not sure, what do the rest of you
think?” versus
“obviously good docstring fix, merged thanks”), and that does
seem to work
ok I think? -njs
-
It might also help to think about what “makes a good committer”.
-
Nathaniel suggests: the defining feature of a good committer is
NOT AT ALL that they are an awesome brilliant coder. The
defining feature
of a good committer is that you can trust them to know their
limits, follow
the project rules, and ask for help or abstain when they’re
unsure. So a
brilliant coder who spits out perfectly-implemented new
features every day
but can’t be bothered to write unit tests should not get
commit rights; a
newbie who knows they’re a newbie is totally fine. I.e. this is not a
position of power, it is a position of responsibility.
Next release
-----------------
not discussed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20150423/2ca58b3a/attachment.html>
More information about the SciPy-Dev
mailing list