[Numpy-discussion] Raveling, reshape order keyword unnecessarily confuses index and memory ordering

[Matthew Brett]

Hi,

On Sat, Mar 30, 2013 at 7:02 PM,  <josef.pktd at gmail.com> wrote:
> On Sat, Mar 30, 2013 at 8:29 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On Sat, Mar 30, 2013 at 7:50 PM,  <josef.pktd at gmail.com> wrote:
>>> On Sat, Mar 30, 2013 at 7:31 PM, Bradley M. Froehle
>>> <brad.froehle at gmail.com> wrote:
>>>> On Sat, Mar 30, 2013 at 3:21 PM, Matthew Brett <matthew.brett at gmail.com>
>>>> wrote:
>>>>>
>>>>> On Sat, Mar 30, 2013 at 2:20 PM,  <josef.pktd at gmail.com> wrote:
>>>>> > On Sat, Mar 30, 2013 at 4:57 PM,  <josef.pktd at gmail.com> wrote:
>>>>> >> On Sat, Mar 30, 2013 at 3:51 PM, Matthew Brett
>>>>> >> <matthew.brett at gmail.com> wrote:
>>>>> >>> On Sat, Mar 30, 2013 at 4:14 AM,  <josef.pktd at gmail.com> wrote:
>>>>> >>>> On Fri, Mar 29, 2013 at 10:08 PM, Matthew Brett
>>>>> >>>> <matthew.brett at gmail.com> wrote:
>>>>> >>>>>
>>>>> >>>>> Ravel and reshape use the tems 'C' and 'F" in the sense of index
>>>>> >>>>> ordering.
>>>>> >>>>>
>>>>> >>>>> This is very confusing.  We think the index ordering and memory
>>>>> >>>>> ordering ideas need to be separated, and specifically, we should
>>>>> >>>>> avoid
>>>>> >>>>> using "C" and "F" to refer to index ordering.
>>>>> >>>>>
>>>>> >>>>> Proposal
>>>>> >>>>> -------------
>>>>> >>>>>
>>>>> >>>>> * Deprecate the use of "C" and "F" meaning backwards and forwards
>>>>> >>>>> index ordering for ravel, reshape
>>>>> >>>>> * Prefer "Z" and "N", being graphical representations of unraveling
>>>>> >>>>> in
>>>>> >>>>> 2 dimensions, axis1 first and axis0 first respectively (excellent
>>>>> >>>>> naming idea by Paul Ivanov)
>>>>> >>>>>
>>>>> >>>>> What do y'all think?
>>>>> >>>>
>>>>> >>>> I always thought "F" and "C" are easy to understand, I always thought
>>>>> >>>> about
>>>>> >>>> the content and never about the memory when using it.
>>>>> >>
>>>>> >> changing the names doesn't make it easier to understand.
>>>>> >> I think the confusion is because the new A and K refer to existing
>>>>> >> memory
>>>>> >>
>>>>>
>>>>> I disagree, I think it's confusing, but I have evidence, and that is
>>>>> that four out of four of us tested ourselves and got it wrong.
>>>>>
>>>>> Perhaps we are particularly dumb or poorly informed, but I think it's
>>>>> rash to assert there is no problem here.
>>>
>>> I think you are overcomplicating things or phrased it as a "trick question"
>>
>> I don't know what you mean by trick question - was there something
>> over-complicated in the example?  I deliberately didn't include
>> various much more confusing examples in "reshape".
>
> I meant making the "candidates" think about memory instead of just
> column versus row stacking.

To be specific, we were teaching about reshaping a (I, J, K, N) 4D
array, it was an image, with time as the 4th dimension (N time
points).   Raveling and reshaping 3D and 4D arrays is a common thing
to do in neuroimaging, as you can imagine.

A student asked what he would get back from raveling this array, a
concatenated time series, or something spatial?

We showed (I'd worked it out by this time) that the first N values
were the time series given by [0, 0, 0, :].

He said - "Oh - I see - so the data is stored as a whole lot of time
series one by one, I thought it would be stored as a series of
images'.

Ironically, this was a Fortran-ordered array in memory, and he was wrong.

So, I think the idea of memory ordering and index ordering is very
easy to confuse, and comes up naturally.

I would like, as a teacher, to be able to say something like:

This is what C memory layout is (it's the memory layout  that gives
arr.flags.C_CONTIGUOUS=True)
This is what F memory layout is (it's the memory layout  that gives
arr.flags.F_CONTIGUOUS=True)
It's rather easy to get something that is neither C or F memory layout
Numpy does many memory layouts.
Ravel and reshape and numpy in general do not care (normally) about C
or F layouts, they only care about index ordering.

My point, that I'm repeating, is that my job is made harder by
'arr.ravel('F')'.

Cheers,

Matthew