[Tutor] memory consumption
Andre' Walker-Loud
walksloud at gmail.com
Wed Jul 3 20:17:53 CEST 2013
Hi All,
I wrote some code that is running out of memory. It involves a set of three nested loops, manipulating a data file (array) of dimension ~ 300 x 256 x 1 x 2. It uses some third party software, but my guess is I am just not aware of how to use proper memory management and it is not the 3rd party software that is the culprit.
Memory management is new to me, and so I am looking for some general guidance. I had assumed that reusing a variable name in a loop would automatically flush the memory by just overwriting it. But this is probably wrong. Below is a very generic version of what I am doing. I hope there is something obvious I am doing wrong or not doing which I can to dump the memory in each cycle of the innermost loop. Hopefully, what I have below is meaningful enough, but again, I am new to this, so we shall see.
################################################
# generic code skeleton
# import a class I wrote to utilize the 3rd party software
import my_class
# instantiate the function do_stuff
my_func = my_class.do_stuff()
# I am manipulating a data array of size ~ 300 x 256 x 1 x 2
data = my_data # my_data is imported just once and has the size above
# instantiate a 3d array of size 20 x 10 x 10 and fill it with all zeros
my_array = numpy.zeros([20,10,10])
# loop over parameters and fill array with desired output
for i in range(loop_1):
for j in range(loop_2):
for k in range(loop_3):
# create tmp_data that has a shape which is the same as data except the first dimension can range from 1 - 1024 instead of being fixed at 300
''' Is the next line where I am causing memory problems? '''
tmp_data = my_class.chop_data(data,i,j,k)
my_func(tmp_data)
my_func.third_party_function()
my_array([i,j,k]) = my_func.results() # this is just a floating point number
''' should I do something to flush tmp_data? '''
#############################################
Thanks,
Andre
More information about the Tutor
mailing list