[Numpy-discussion] histogramdd memory needs

[Lars Friedrich]

Hi,

> 2) Is there a way to use another algorithm (at the cost of performance)
>> > that uses less memory during calculation so that I can generate bigger
>> > histograms?
> 
> 
> You could work through your array block by block. Simply fix the range and
> generate an histogram for each slice of 100k data and sum them up at the
> end.

Thank you for your answer.

I sliced the (original) data into blocks. However, when I do this, I 
need at least twice the memory for the whole histogram (one for the 
temporary result and one for accumulating the total result). Assuming my 
histogram has a size of (280**3)*8 = 176 (megabytes) this does not help, 
I think.

What I will try next is to compute smaller parts of the big histogram 
and combine them at the end. (Slice the histogram into blocks) Is it 
this, that you were recommending?

Lars