Large data arrays?

[Ole Streicher]

Hi John

John Machin <sjmachin at lexicon.net> writes:
> On Apr 25, 1:14 am, Ole Streicher <ole-usenet-s... at gmx.net> wrote:
>> John Machin <sjmac... at lexicon.net> writes:
>> >> From my access pattern, it would be probably better to combine 25 rows
>> >> into one slice and have one matrix where every cell contains 25 rows.
>> >> Are there any objections about that?
>> > Can't object, because I'm not sure what you mean ... how many elements
>> > in a "cell"?
>>
>> Well, a matrix consists of "cells"? A 10x10 matrix has 100 "cells".
>
> Yes yes but you said "every cell contains 25 rows" ... what's in a
> cell? 25 rows, with each row containing what?

I mean: original cells.
I have 100.000x4096 entries:

(0,0) (0,1) ... (0,4095)
(1,0) (1,1) ... (1,4095)
...
(100.000,0) (100.000,1) ... (100.000,4095)

This will be re-organized in a new matrix, containing 4096 columns (as
before) and 4000 rows. The leftmost cell (first row, first col) in the
new matrix then contains the array

(0,0)
(1,0)
...
(24,0)

The second column of the first row contains the array

(0,1)
(1,1)
...
(24,1)

and so on. The first column of the second row contains

(25,0)
...
(49,0)

That way, I get a new matrix where every cell contains an array of 24
"original" cells. Disadvantage (what I see now when I write it down)
is that this is bad for numpy since it deals with arrays instead of
numbers in matrix positions.

Best regards

Ole