[Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 4:18 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, Oct 29, 2011 at 5:11 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett <matthew.brett at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>> >> <ralf.gommers at googlemail.com> wrote:
>> >> >
>> >> >
>> >> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
>> >> > <matthew.brett at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
>> >> >> <matthew.brett at gmail.com>
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>> >> >> > <ralf.gommers at googlemail.com> wrote:
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>> >> >> >> <matthew.brett at gmail.com>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >> >> >>> <ralf.gommers at googlemail.com> wrote:
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>> >> >> >>> > <matthew.brett at gmail.com>
>> >> >> >>> > wrote:
>> >> >> >>> >>
>> >> >> >>> >> Hi,
>> >> >> >>> >>
>> >> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >> >> >>> >> <ralf.gommers at googlemail.com> wrote:
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >> >> >>> >> > <matthew.brett at gmail.com>
>> >> >> >>> >> > wrote:
>> >> >> >>> >> >>
>> >> >> >>> >> >> Hi,
>> >> >> >>> >> >>
>> >> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >> >> >>> >> >> <charlesr.harris at gmail.com> wrote:
>> >> >> >>> >> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>> >> >> >>> >> >> Nathaniel
>> >> >> >>> >> >> was
>> >> >> >>> >> >> pointing to links for projects that care that everyone
>> >> >> >>> >> >> agrees
>> >> >> >>> >> >> before
>> >> >> >>> >> >> they go ahead.
>> >> >> >>> >> >
>> >> >> >>> >> > It looked to me like there was a serious intent to come to
>> >> >> >>> >> > an
>> >> >> >>> >> > agreement,
>> >> >> >>> >> > or
>> >> >> >>> >> > at least closer together. The discussion in the summer was
>> >> >> >>> >> > going
>> >> >> >>> >> > around
>> >> >> >>> >> > in
>> >> >> >>> >> > circles though, and was too abstract and complex to follow.
>> >> >> >>> >> > Therefore
>> >> >> >>> >> > Mark's
>> >> >> >>> >> > choice of implementing something and then asking for
>> >> >> >>> >> > feedback
>> >> >> >>> >> > made
>> >> >> >>> >> > sense
>> >> >> >>> >> > to
>> >> >> >>> >> > me.
>> >> >> >>> >>
>> >> >> >>> >> I should point out that the implementation hasn't - as far as
>> >> >> >>> >> I
>> >> >> >>> >> can
>> >> >> >>> >> see - changed the discussion.  The discussion was about the
>> >> >> >>> >> API.
>> >> >> >>> >>
>> >> >> >>> >> Implementations are useful for agreed APIs because they can
>> >> >> >>> >> point
>> >> >> >>> >> out
>> >> >> >>> >> where the API does not make sense or cannot be implemented.
>> >> >> >>> >>  In
>> >> >> >>> >> this
>> >> >> >>> >> case, the API Mark said he was going to implement - he did
>> >> >> >>> >> implement -
>> >> >> >>> >> at least as far as I can see.  Again, I'm happy to be
>> >> >> >>> >> corrected.
>> >> >> >>> >
>> >> >> >>> > Implementations can also help the discussion along, by
>> >> >> >>> > allowing
>> >> >> >>> > people
>> >> >> >>> > to
>> >> >> >>> > try out some of the proposed changes. It also allows to
>> >> >> >>> > construct
>> >> >> >>> > examples
>> >> >> >>> > that show weaknesses, possibly to be solved by an alternative
>> >> >> >>> > API.
>> >> >> >>> > Maybe
>> >> >> >>> > you
>> >> >> >>> > can hold the complete history of this topic in your head and
>> >> >> >>> > comprehend
>> >> >> >>> > it,
>> >> >> >>> > but for me it would be very helpful if someone said:
>> >> >> >>> > - here's my dataset
>> >> >> >>> > - this is what I want to do with it
>> >> >> >>> > - this is the best I can do with the current implementation
>> >> >> >>> > - here's how API X would allow me to solve this better or
>> >> >> >>> > simpler
>> >> >> >>> > This can be done much better with actual data and an actual
>> >> >> >>> > implementation
>> >> >> >>> > than with a design proposal. You seem to disagree with this
>> >> >> >>> > statement.
>> >> >> >>> > That's fine. I would hope though that you recognize that
>> >> >> >>> > concrete
>> >> >> >>> > examples
>> >> >> >>> > help people like me, and construct one or two to help us out.
>> >> >> >>> That's what use-cases are for in designing APIs.  There are
>> >> >> >>> examples
>> >> >> >>> of use in the NEP:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >> >> >>>
>> >> >> >>> the alterNEP:
>> >> >> >>>
>> >> >> >>> https://gist.github.com/1056379
>> >> >> >>>
>> >> >> >>> and my longer email to Travis:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>> >> >> >>>
>> >> >> >>> Mark has done a nice job of documentation:
>> >> >> >>>
>> >> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>> >> >> >>>
>> >> >> >>> If you want to understand what the alterNEP case is, I'd suggest
>> >> >> >>> the
>> >> >> >>> email, just because it's the most recent and I think the
>> >> >> >>> terminology
>> >> >> >>> is slightly clearer.
>> >> >> >>>
>> >> >> >>> Doing the same examples on a larger array won't make the point
>> >> >> >>> easier
>> >> >> >>> to understand.  The discussion is about what the right concepts
>> >> >> >>> are,
>> >> >> >>> and you can help by looking at the snippets of code in those
>> >> >> >>> documents, and deciding for yourself whether you think the
>> >> >> >>> current
>> >> >> >>> masking / NA implementation seems natural and easy to explain,
>> >> >> >>> or
>> >> >> >>> rather forced and difficult to explain, and then email back
>> >> >> >>> trying
>> >> >> >>> to
>> >> >> >>> explain your impression (which is not always easy).
>> >> >> >>
>> >> >> >> If you seriously believe that looking at a few snippets is as
>> >> >> >> helpful
>> >> >> >> and
>> >> >> >> instructive as being able to play around with them in IPython and
>> >> >> >> modify
>> >> >> >> them, then I guess we won't make progress in this part of the
>> >> >> >> discussion.
>> >> >> >> You're just telling me to go back and re-read things I'd already
>> >> >> >> read.
>> >> >> >
>> >> >> > The snippets are in ipython or doctest format - aren't they?
>> >> >>
>> >> >> Oops - 10 minute rule.  Now I see that you mean that you can't
>> >> >> experiment with the alternative implementation without working code.
>> >> >
>> >> > Indeed.
>> >> >
>> >> >>
>> >> >> That's true, but I am hoping that the difference between - say:
>> >> >>
>> >> >> a[0:2] = np.NA
>> >> >>
>> >> >> and
>> >> >>
>> >> >> a.mask[0:2] = False
>> >> >>
>> >> >> would be easy enough to imagine.
>> >> >
>> >> > It is in this case. I agree the explicit ``a.mask`` is clearer. This
>> >> > is
>> >> > a
>> >> > quite specific point that could be improved in the current
>> >> > implementation.
>> >>
>> >> Thanks - this is helpful.
>> >>
>> >> > It doesn't require ripping everything out.
>> >>
>> >> Nathaniel wasn't proposing 'ripping everything out' - but backing off
>> >> until consensus has been reached.  That's different.    If you think
>> >> we should not do that, and you are interested, please say why.
>> >> Second - I was proposing that we do indeed keep the code in the
>> >> codebase but discuss adaptations that could achieve consensus.
>> >>
>> >
>> > I'm much opposed to ripping the current code out.
>>
>> You are repeating the loaded phrase 'ripping the current code out' and
>> thus making the discussion less sensible and more hostile.
>>
>> >  It isn't like it is (known
>> > to be) buggy, nor has anyone made the case that it isn't a basis on
>> > which
>> > build other options. It also smacks of gratuitous violence committed by
>> > someone yet to make a positive contribution to the project.
>>
>> This is cheap, rude, and silly.  All I can see from Nathaniel is a
>> reasonable, fair attempt to discuss the code.  He proposed backing off
>> the code in good faith.   You are emphatically, and, in my view
>> childishly, ignoring the substantial points he is making, and
>> asserting over and over that he deserves no hearing because he has not
>> contributed code.
>
> Sorry Matthew, but Nathaniel's interaction comes across to me as arrogant,
> and your constant use of terms like childish, destructive to the community,
> etc. come across as manipulative.

I don't know what 'manipulative' means here.  Can you explain?

> I can live with the words, but you aren't
> doing much to get this developer on your side.

No, I am not trying to get you on my side because I don't believe in
sides, and, unless you tell me otherwise, I think you believe in a
implicit model of decision making that is bad for numpy.  I will
willingly and enthusiastically buy you a drink at scipy - but I
believe you are wrong in the way that you have approached this
discussion, and I believe the model that you are using, and it's
opposite - are of central importance to the health of our shared
discussions in the future.

Best,

Matthew