[Numpy-discussion] alterNEP - was: missing data discussion round 2

[Mark Wiebe]

On Fri, Jul 1, 2011 at 6:58 AM, Matthew Brett <matthew.brett at gmail.com>wrote:

> Hi,
>
> On Fri, Jul 1, 2011 at 2:36 AM, Keith Goodman <kwgoodman at gmail.com> wrote:
> > On Thu, Jun 30, 2011 at 10:51 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >> On Thu, Jun 30, 2011 at 6:31 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> >>> In the interest of making the discussion as concrete as possible, here
> >>> is my draft of an alternative proposal for NAs and masking, based on
> >>> Nathaniel's comments.  Writing it, it seemed to me that Nathaniel is
> >>> right, that the ideas become much clearer when the NA idea and the
> >>> MASK idea are separate.   Please do pitch in for things I may have
> >>> missed or misunderstood:
> >> [...]
> >>
> >> Thanks for writing this up! I stuck it up as a gist so we can edit it
> >> more easily:
> >>  https://gist.github.com/1056379/
> >> This is your initial version:
> >>
> https://gist.github.com/1056379/c809715f4e9765db72908c605468304ea1eb2191
> >> And I made a few changes:
> >>
> https://gist.github.com/1056379/33ba20300e1b72156c8fb655bd1ceef03f8a6583
> >> Specifically, I added a rationale section, changed np.MASKED to
> >> np.IGNORE (as per comments in this thread), and added a vowel to
> >> "propmsk".
> >
> > It might be helpful to make a small toy class in python so that people
> > can play around with NA and IGNORE from the alterNEP.
>
> Thanks for doing this.
>
> I don't know about you, but I don't know where to work on the
> discussion or draft implementation, because I am not sure where the
> disagreement is.  Lluis has helpfully pointed out a specific case of
> interest.   Pierre has fed back with some points of clarification.
> However, other than that, I'm not sure what we should be discussing.
>
> @Mark
> @Chuck
> @anyone
>
> Do you see problems with the alterNEP proposal?


Yes, I really like my design as it stands now, and the alterNEP removes a
lot of the abstraction and interoperability that are in my opinion the best
parts. I've made more updates to the NEP based on continuing feedback, which
are part of the pull request I want reviews for.


> If so, what are they?
>

Mainly: Reduced interoperability, more complex implementation (leading to
more bugs), and an unclear theoretical model for the masked part of it.


> Do you agree that the alterNEP proposal is easier to understand?


No.

If not, can you explain why?
>

My answers to that are already scattered in the emails in various places,
and in the various rationales and justifications provided in the NEP.


> What do you see as the important points of difference between the NEP
> and the alterNEP?
>

The biggest thing is the NEP supports more use cases in a clean way by
composition of different simpler components. It defines one clear missing
data abstraction, and proposes two implementations that are interchangeable
and can interoperate. The alterNEP proposes two independent APIs, reducing
interoperability and so significantly increasing the amount of learning
required to work with both of them. This also precludes switching between
the two approaches without a lot of work.

The current pull request that's sitting there waiting for review does not
have an impact on which approach goes ahead, but the code I'm doing now
does. This is a fairly large project, and I don't have a great length of
time to do it in, so I'm not going to participate extensively in the
alterNEP discussion. If you want to help me, please review my code and
provide specific feedback on my NEP (the code review system in github is
great for this too, I've received some excellent feedback on the NEP that
way). If you want to change my mind about things, please address the
specific design decisions you think are problematic by specifically
responding to lines in the NEP, as part of code-reviewing my pull request in
github.

Thanks,
-Mark

@Pierre - what do you think?
>
> Best,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110701/b2740362/attachment.html>