[Numpy-discussion] in the NA discussion, what can we agree on?

[Gary Strangman]

>> destructive + propagating = the data point is truly missing (satellite fell into
>> the ocean; dog ate my source datasheet, or whatever), this is the nature of that
>> data point, such missingness should be replicated in elementwise operations, and
>> the missingness SHOULD interfere with reduction operations that involve that
>> datapoint (np.sum([1,MISSING])=MISSING)
>
> Right.
>
>
>> destructive + non-propagating = the data point is truly missing, this is the
>> nature of that data point, such missingness should be replicated in elementwise
>> operations, but such missingness should NOT interfere with reduction operations
>> that involve that datapoint (np.sum([1,MISSING])=1)
>
> What do you define as element-wise operations?
>
> Is a sum on an array an element-wise operation?
>
>  >>> [1, MISSING]+2
>  [1, MISSING]
>
> Or is it just a form of reduction (after shape broadcasting)?
>
>  >>> [1, MISSING]+2
>  [3, 2]
>
> For me it's the second, so the only time where special values "propagate" in a
> non-propagating scenario is when you slice an array.

Let's say I want to re-scale a column (or remove the mean from a column). 
I wouldn't want that to change my "missingness". Thus, I'm thinking:

>>> x = [1,2,MISSING]
>>> x*3
[3, 6, MISSING]
>>> x = [1,2,MISSING]
>>> x - x.mean()
[-0.5, 0.5, MISSING]

To me it makes sense to have identical operations for the temporary IGNORE 
case below (versus the permanent MISSING case here). Note, the reason to 
independently have separate IGNORE and MISSING is so that I can (for 
example) temporarily IGNORE entire rows in my 2D array (which may have 
scattered MISSING elements), and when I undo the IGNORE operation the 
MISSING elements are still MISSING.

The question does still remain what to do when performing operations like 
those above in IGNORE cases. Perform the operation "underneath"? Or not?

>> non-destructive + propagating = I want to ignore this datapoint for now;
>> element-wise operations should replicate this "ignore" designation, and
>> missingness of this type SHOULD interfere with reduction operations that involve
>> this datapoint (np.sum([1,IGNORE])=IGNORE)
>
> Right.
>
>
>> non-destructive + non-propagating = I want to ignore this datapoint for now;
>> element-wise operations should replicate this "ignore" designation, but
>> missingness of this type SHOULD NOT interfere with reduction operations that
>> involve this datapoint (np.sum([1,IGNORE])=1)
>
> Same concerns as above.
>
>
> Lluis
>
>


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.