[Numpy-discussion] Raveling, reshape order keyword unnecessarily confuses index and memory ordering

[Matthew Brett]

Hi,

On Fri, Apr 5, 2013 at 3:09 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Fri, Apr 5, 2013 at 12:53 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>
>>
>> On Fri, Apr 5, 2013 at 9:21 PM, Matthew Brett <matthew.brett at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> On Fri, Apr 5, 2013 at 3:09 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>> >
>>> >
>>> >
>>> > On Fri, Apr 5, 2013 at 5:13 PM, Matthew Brett <matthew.brett at gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> On Fri, Apr 5, 2013 at 2:20 AM, Sebastian Berg
>>> >> <sebastian at sipsolutions.net> wrote:
>>> >> > Hey
>>> >> >
>>> >> > On Thu, 2013-04-04 at 14:20 -0700, Matthew Brett wrote:
>>> >> >> Hi,
>>> >> >>
>>> >> >> On Tue, Apr 2, 2013 at 4:32 AM, Nathaniel Smith <njs at pobox.com>
>>> >> >> wrote:
>>> >> >> <snip>
>>> >> >> > Maybe we should go through and rename "order" to something more
>>> >> >> > descriptive
>>> >> >> > in each case, so we'd have
>>> >> >> >   a.reshape(..., index_order="C")
>>> >> >> >   a.copy(memory_order="F")
>>> >> >> > etc.?
>>> >> >>
>>> >> >> I'd like to propose this instead:
>>> >> >>
>>> >> >> a.reshape(..., order="C")
>>> >> >> a.copy(layout="F")
>>> >> >>
>>> >> >
>>> >> > I actually like this, makes the point clearer that it has to do with
>>> >> > memory layout and implies contiguity, plus it is short and from the
>>> >> > numpy perspective copy, etc. are the ones that add additional info to
>>> >> > "order" and not reshape (because IMO memory order is something new
>>> >> > users
>>> >> > should not worry about at first). A and K orders will still have
>>> >> > their
>>> >> > quirks with np.array and copy=True/False, but for many functions they
>>> >> > are esoteric anyway.
>>> >> >
>>> >> > It will be one hell of a deprecation though, but I am +0.5 for adding
>>> >> > an
>>> >> > alias for now (maybe someone knows an even better name?), but I think
>>> >> > that in this case, it probably really is better to wait with actual
>>> >> > deprecation warnings for a few versions, since it touches a *lot* of
>>> >> > code. Plus I think at the point of starting deprecation warnings (and
>>> >> > best earlier) numpy should provide an automatic fixer script...
>>> >> >
>>> >> > The only counter point that remains for me is the difficulty of
>>> >> > deprecation, since I think the new name idea is very clean. And this
>>> >> > is
>>> >> > unfortunately even more invasive then the index_order proposal.
>>> >>
>>> >> I completely agree that we'd have to be gentle with the change.  The
>>> >> problem we'd want to avoid is people innocently using 'layout' and
>>> >> finding to their annoyance that the code doesn't work with other
>>> >> people's numpy.
>>> >>
>>> >> How about:
>>> >>
>>> >> Step 1:  'order' remains as named keyword, layout added as alias,
>>> >> comment on the lines of "layout will become the default keyword for
>>> >> this option in later versions of numpy; please consider updating any
>>> >> code that does not need to remain backwards compatible'.
>>> >>
>>> >> Step 2: default keyword becomes 'layout' with 'order' as alias,
>>> >> comment like "order is an alias for 'layout' to maintain backwards
>>> >> compatibility with numpy <= 1.7.1', please update any code that does
>>> >> not need to maintain backwards compatibility with these numpy
>>> >> versions'
>>> >>
>>> >> Step 3: Add deprecation warning for 'order', "order will be removed as
>>> >> an alias in future versions of numpy"
>>> >>
>>> >> Step 4: (distant future) Remove alias
>>> >>
>>> >> ?
>>> >
>>> >
>>> > A very strong -1 from me. Now we're talking about deprecation warnings
>>> > and a
>>> > backwards compatibility break after all. I thought we agreed that this
>>> > was a
>>> > very bad idea, so why are you proposing it now?
>>> >
>>> > Here's how I see it: deprecation of "order" is a no go. Therefore we
>>> > have
>>> > two choices here:
>>> > 1. Simply document the current "order" keyword better and leave it at
>>> > that.
>>> > 2. Add a "layout" (or "index_order") keyword, and live with both "order"
>>> > and
>>> > "layout" keywords forever.
>>> >
>>> > (2) is at least as confusing as (1), more work and poor design.
>>> > Therefore I
>>> > propose to go with (1).
>>>
>>> You are saying that deprecation of 'order' at any stage in the next 10
>>> years of numpy's lifetime is a no go?
>>
>>
>> For something like this? Yes.
>
> You are saying I think that I am wrong in thinking this is an
> important change that will make numpy easier to explain and use in the
> long term.
>
> You'd probably expect me to disagree, and I do.  I think I am right in
> thinking the change is important - I've tried to make that case in
> this thread, as well as I can.
>
>>> I think that is short-sighted and I think it will damage numpy.
>>
>>
>> It will damage numpy to be conservative and not change a name for a little
>> bit of clarity for some people that avoids reading the docs maybe a little
>> more carefully? There's a lot of things that can damage numpy, but this
>> isn't even close in my book. Too few developers, continuous backwards
>> compatibility issues, faster alternative libraries surpassing numpy - that's
>> the kind of thing that causes damage.
>
> We're talked about consensus on this list.  Of course it can be very
> hard to achieve.
>
>>> Believe me, I have as much investment in backward compatibility as you
>>> do.  All the three libraries that I spend a long time maintaining need
>>> to test against old numpy versions - but - for heaven's sake - only
>>> back to numpy 1.2 or numpy 1.3.  We don't support Python 2.5 any more,
>>> and I don't think we need to maintain compatibility with Numeric
>>> either.
>>
>>
>> Really? This is from 3 months ago:
>> http://article.gmane.org/gmane.comp.python.numeric.general/52632. It's now
>> 2013, we are probably dropping numarray compat in 1.8. Not exactly 10 years,
>> but of the same order.
>
> I am happy to make this change over the same time course if you think
> that is necessary.

I am also happy with only steps 1 and 2 if you feel that deprecation
over any time scale is unacceptable.

Best,

Matthew