[Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)

[Matthew Brett]

Hi,

Thanks for replying, that is helpful.

I'm going to write carefully, because I suspect we've only got this
one shot at getting it right.

I first wanted to come back to an earlier email of yours:

On Thu, Aug 27, 2015 at 3:05 AM, Nathaniel Smith <njs at pobox.com> wrote:
> At least personally, I have to admit that the idea of a governance
> model involving elections fills me with creeping horror. The reason is
> that whole point of having a governance model (IMHO) is to (a)
> minimize the rise of interpersonal drama, (b) when some amount of
> interpersonal drama does inevitably arise anyway, provide some
> regulated channel for it, hopefully one that leads to a drama sink.

I think this is the wrong way to think of the problem.   There is no drama when:

* everyone agrees (problem is trivial);
* no one cares (problem is not important);
* everyone who does care has come to the conclusion that it is not
worth arguing.

The opposite of drama is not efficiency, but boredom and frustration.

As evidence, vigorous, substantial and bad-tempered debates are
characteristic of successful companies:

"""
"They were chaos. We would stay there for hours, ironing out the
issues, until we came to something ... At times the meetings would get
so violent that people almost went across the table at each other....
People yelled. They waved their arms around and pounded on tables.
Faces would get red and veins bulged out".
""" [1]

I'm sure you have hung out with the IPython team as I have.  You will
then hear their description of how much the team disagree with each
other, and often, how important that disagreement is, to finding the
right answer.

On the other hand, I don't think it is our culture in numpy to value
that kind of debate.  That brings us to my next comment:

On Fri, Sep 4, 2015 at 8:35 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río
>> <jaime.frio at gmail.com> wrote:
> [...]
>>> <joke>
>>> Are you trying to prove the point that consensus doesn't work by making it
>>> impossible to reach a consensus on this? ;-)
>>> </joke>
>>
>> Forgive me if I use this joke to see if I can get us any further.
>>
>> If this was code, I think this joke would not be funny, because we
>> wouldn't expect to reach consensus without considering all the
>> options, and discussing their pros and cons.
>>
>> Why would that not be useful in the case of forms of governance?
>>
>> One reason might be that the specific form of governance can have no
>> influence on the long-term health of the project.
>>
>> I am convinced that that is wrong - that the form of governance has a
>> large influence on the long-term health of a project.
>>
>> If there is some possibility that this is true, then it seems to me
>> that we would be foolish not to try and come to some reasoned choice
>> about the form of governance.
>
> That seems fair, and I also think that even if we end up not changing
> anything then there's still some utility in having the discussion.
> Certainly one of my secret fears in writing all this up is that it
> would end up being just some words that only I cared about and
> everyone else ignored, so seeing other people engaging with it is
> heartening :-).
>
> I'm not feeling inspired this Friday evening to write a full white
> paper reviewing the universe of possible governance, but I'll
> summarize a bit what the reasoning was that led us to what we ended up
> with (at least as I remember it and from my perspective).
>
> Some of the sources that were mentioned for reference and positive or
> negative inspiration include Karl Fogel's chapter on governance, the
> Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc
> ditto, subversion, Jupyter/ipython, the Linux kernel, python,
> matplotlib, the Apache foundation's processes, the gnome foundation,
> nodejs, ... I'm certainly forgetting some.
>
> The basic observation that we started from was that all the successful
> projects seemed to fit a similar template:
> (1) basically all actual decision making is done by informal and
> maximally inclusive consensus-based processes,
> (2) there are some sort of more formal "backstop" rules that are
> called into play if informal processes break down (which hopefully
> never even happens).
> These projects don't tend to look that similar on paper, because
> people tend to only write down the "backstop" rules, and those vary a
> lot.
>
> So e.g., for some projects like Python or IPython, the formal rule is
> "the BDFL decides". But then if you look at what successful BDFLs do
> with this power, they mostly enforce rule #1 above. (Karl discusses
> this: http://producingoss.com/en/producingoss.html#benevolent-dictator).
> E.g., if you watch Guido, I don't think I've ever seen him say "I know
> people don't like X but we're doing it anyway". He mostly sits back to
> watch the debate, steps in occasionally to prune off unproductive
> lines of conversation (e.g. in the PEP 465 debate where he watched for
> a while, stepped in to tell people to knock it off with the pointless
> bikeshedding about which character to use, and then went back to
> watching) or encourage people to keep thinking about something, is
> always very careful not to actually make any strong statements, etc.
> Or, e.g., the bit in the draft governance about how the steering
> committee's secondary job is to make decisions, and their primary job
> is to prevent things reaching the point where they have to? That's a
> straight paraphrase of Fernando talking about his BDFL philosophy.
>
> Or on the other end of things, you have e.g. Subversion, which had an
> elaborate defined governance system with different levels of
> "core-ness", a voting system, etc. -- and they were 6 years into the
> project before they had their first vote. (The vote was on the crucial
> technical decision of whether to write function calls like "f ()" or
> "f()".) Or e.g. the massive and famously fractious Debian has their
> CTTE, which theoretically is the final arbiter of technical decisions
> in Debian and has immense powers... but they issue like ~5 rulings a
> year on average, and have gone years without issuing a ruling at all.
>
> So, our thought process was: for our purposes, since we don't have a
> BDFL, we can't depend on them to be the unwritten mechanism for
> enforcing rule #1, so first we should write down rule #1.

What you are describing in the previous paragraph seems to be a
well-known feature of successful leaders.  The CEOs of highly
successful companies described in [2] differed from less-successful
comparators in encouraging debate and acting as the final arbiter for
the decisions of the group.

I don't think you are right that you can create that effect without
having such a leader.   The point about the leader is that they are a
trusted referee and mentor to the group.   They provide stability and
space for the debate.  For example, in my little church, there is an
elected position called the 'moderator' with that purpose.  Most
groups recognize that someone has to do that job, and that the group
cannot do that themselves.

On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith <njs at pobox.com> wrote:
> I guess I just don't see how you think we can legislate ourselves into
> having a vision. Like... we'll elect a volunteer to produce a roadmap,
> and then if they don't we'll fire them (accountability!), leaving us
> with no volunteer and no roadmap? How will that help?

I agree it is not sensible for the president or moderator to produce a
roadmap, but a roadmap is different from a vision.

We often run into trouble on the list, discussing whether to make
backwards-incompatible changes.   The reason for the trouble is that
different people have different ideas about what numpy is.  Some
people think numpy is a legacy library who's job it is to tread water
while waiting for a better alternative.  For them, backwards
compatibility is of overriding importance.  Others think that numpy
should be gradually refactored into a better and more extensible
version that will continue to be the backbone of scientific computing
into the fairly distant future.  These are two different visions that
are not compatible one with another.  Trying to keep both groups happy
is likely to lead to dissatisfaction and frustration from both sides.

On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith <njs at pobox.com> wrote:
> I'm not sure how useful this focus on NetBSD is, given the elephant in
> the room: the fundamental challenge for NetBSD is that NetBSD has no
> compelling reason to exist;

That is now true, but I would argue (I think Hannum would argue) that
this was the direct effect of very poor management, with no vision for
where the project was going.  I imagine you'd agree that OpenBSD does
have a purpose, and OpenBSD started as a one-person-in-charge fork of
NetBSD, where the fork was the result of very bad management by the
'core'.

 I hate to quote these again but:

"[the "core" and "board of directors"] teams are dysfunctional because
they do not provide leadership: all they do is act reactively to
requests from users and/or to resolve internal disputes. In other
words: there is no initiative nor vision emerging from these teams
(and, for that matter, from anybody)."

"As things are today, the project leaders will almost always settle
for either the conservative solution or a solution that compromises on
every detail so as to not annoy anyone. " [3]

A organization that is governed like this, just has to fail, in the long term.

Cheers,

Matthew

[1] 'Exhibit E' in
http://nipyworld.blogspot.co.uk/2012/02/what-is-meritocracy-in-open-source.html
[2] https://en.wikipedia.org/wiki/Good_to_Great
[3] http://asterisk.dynevor.org/melting-core.html