[Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)

[Travis Oliphant]

On Apr 16, 2012, at 8:03 PM, Matthew Brett wrote:

> Hi,
> 
> On Mon, Apr 16, 2012 at 3:06 PM, Travis Oliphant <travis at continuum.io> wrote:
> 
>> I have heard from a few people that they are not excited by the growth of
>> the NumPy data-structure by the 3 pointers needed to hold the masked-array
>> storage.   This is especially true when there is talk to potentially add
>> additional attributes to the NumPy array (for labels and other
>> meta-information).      If you are willing to let us know how you feel about
>> this, please speak up.
> 
> I guess there are two questions here
> 
> 1) Will something like the current version of masked arrays have a
> long term future in numpy, regardless of eventual API? Most likely
> answer - yes?

I think the answer to this is yes, but it could be as a feature-filled sub-class (like the current numpy.ma, except in C). 

> 2) Will likely changes to the masked array API make any difference to
> the number of extra pointers?  Likely answer no?
> 
> Is that right?

The answer to this is very likely no on the Python side.  But, on the C-side, their could be some differences (i.e. are masked arrays a sub-class of the ndarray or not). 

> 
> I have the impression that the masked array API discussion still has
> not come out fully into the unforgiving light of discussion day, but
> if the answer to 2) is No, then I suppose the API discussion is not
> relevant to the 3 pointers change.

You are correct that the API discussion is separate from this one.     Overall,  I was surprised at how fervently people would oppose ABI changes.   As has been pointed out, NumPy and Numeric before it were not really designed to prevent having to recompile when changes were made.   I'm still not sure that a better overall solution is not to promote better availability of downstream binary packages than excessively worry about ABI changes in NumPy.    But, that is the current climate. 

In that climate, my concern is that we haven't finalized the API but are rapidly cementing the *structure* of NumPy arrays into a modified form that has real downstream implications.   Two other people I have talked to share this concern (nobody who has posted on this list before but who are heavy users of NumPy).    I may have missed the threads where it was discussed, but have these structure changes and their implications been fully discussed?   Is there anyone else who is concerned about adding 3 more pointers (12 bytes or 24 bytes) to the NumPy structure?     

As Chuck points out, 3 more pointers is not necessarily that big of a deal if you are talking about a large array (though for small arrays it could matter).   But, I personally know of half-written NEPs that propose to add more pointers to the NumPy array: 

	* to allow meta-information to be attached to a NumPy array
	* to allow labels to be attached to a NumPy array (ala data-array)
	* to allow multiple chunks for an array.

Are people O.K. with 5 or 6 more pointers on every NumPy array?    We could also think about adding just one more pointer to a new "enhanced" structure that contains multiple enhancements to the NumPy array.    

But, this whole line of discussion sounds a lot like a true sub-class of the NumPy array at the C-level.    It has the benefit that only people that use the features of the sub-class have to worry about using the extra space.  

Mark and I will talk about this long and hard.  Mark has ideas about where he wants to see NumPy go, but I don't think we have fully accounted for where NumPy and its user base *is* and there may be better ways to approach this evolution.    If others are interested in the outcome of the discussion please speak up (either on the list or privately) and we will make sure your views get heard and accounted for. 

Best regards,

-Travis





> 
> See y'all,
> 
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion