[Numpy-discussion] How to limit the numpy.memmap's RAM usage?

[braingateway]

David Cournapeau :
> 2010/10/23 braingateway <braingateway at gmail.com>:
>   
>> Hi everyone,
>> I noticed the numpy.memmap using RAM to buffer data from memmap files.
>> If I get a 100GB array in a memmap file and process it block by block,
>> the RAM usage is going to increasing with the process running until
>> there is no available space in RAM (4GB), even though the block size is
>> only 1MB.
>> for example:
>> ####
>> a = numpy.memmap(‘a.bin’, dtype='float64', mode='r')
>> blocklen=1e5
>> b=npy.zeros((len(a)/blocklen,))
>> for i in range(0,len(a)/blocklen):
>> b[i]=npy.mean(a[i*blocklen:(i+1)*blocklen])
>> ####
>> Is there any way to restrict the memory usage in numpy.memmap?
>>     
>
> The whole point of using memmap is to let the OS do the buffering for
> you (which is likely to do a better job than you in many cases). Which
> OS are you using ? And how do you measure how much memory is taken by
> numpy for your array ?
>
> David
> _______________________________________________
>   
Hi David,

I agree with you about the point of using memmap. That is why the 
behavior is so strange to me.
I actually measure the size of resident set (pink trace in figure2) of 
the python process on Windows. Here I attached the  result. You can see 
the  RAM  usage is definitely not file system cache.

LittleBigBrain
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpyMemmapAvaRAM.PNG
Type: image/png
Size: 13416 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20101023/16c011eb/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpyMemmapAvaRAM2.PNG
Type: image/png
Size: 18734 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20101023/16c011eb/attachment-0001.png>