[SciPy-User] efficient way to store and use a 4D redundant matrix

[Frédéric Bastien]

If the data is easily compressible maybe you can use carray[1]? It
keep data compressed in memory and allow to make some operation
direction on the compressed data.

Frédéric Bastien

[1] http://pypi.python.org/pypi/carray/0.4

On Fri, Mar 18, 2011 at 6:01 AM, Emanuele Olivetti
<emanuele at relativita.com> wrote:
> On 03/16/2011 08:31 PM, Daniel Lepage wrote:
>> [...]
>>
>> The sparse matrix formats will only help you if you can rewrite A in
>> terms of matrices that are mostly 0.
>>
>
> Correct. This is not my case, you are right.
>
>> Do you need the results of slicing, reshaping, etc. to also be
>> similarly compressed? If so, I can't see any way to implement this
>> without an index array, because once you reshape or slice A you won't
>> know which cells correspond to which indices in the original A.
>>
>
> I will have a deeper look to a solution with index array. Thanks for
> pointing it out.
>
>> If you're only taking small slices of this and then applying linear
>> algebra operations to those, you might be better off writing a class
>> that looks up the relevant values on the fly; you could overload
>> __getitem__ so that e.g. A[:,1,:,3] would generate the correct float64
>> array on the fly and return it.
>>
>
> Unfortunately I am not playing with small slices. So I guess that overloading
> __getitem__ would be impractical.
>
>> However, if the nonredundant part takes only ~4MB, then maybe I don't
>> understand your layout - for a 100x100x100x100 and 64-bit floats, I
>> think the nonredundant part should take ((100 choose 4) + ((100 choose
>> 3) * 3) + ((100 choose 2) * 3) + (100 choose 1)) * 8 bytes = about
>> 34MB. Was that a math error, or am I misunderstanding the question?
>>
>
> My fault. It is indeed ~34Mb. I missed one order of magnitude when
> computing (100**4 * 8byte) / 24 .
>
> Thanks again,
>
> Emanuele
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>