[Tutor] memory consumption

Andre' Walker-Loud walksloud at gmail.com
Wed Jul 3 20:17:53 CEST 2013


Hi All,

I wrote some code that is running out of memory.  It involves a set of three nested loops, manipulating a data file (array) of dimension ~ 300 x 256 x 1 x 2.  It uses some third party software, but my guess is I am just not aware of how to use proper memory management and it is not the 3rd party software that is the culprit.  

Memory management is new to me, and so I am looking for some general guidance.  I had assumed that reusing a variable name in a loop would automatically flush the memory by just overwriting it.  But this is probably wrong.  Below is a very generic version of what I am doing.  I hope there is something obvious I am doing wrong or not doing which I can to dump the memory in each cycle of the innermost loop.  Hopefully, what I have below is meaningful enough, but again, I am new to this, so we shall see.

################################################
# generic code skeleton
# import a class I wrote to utilize the 3rd party software
import my_class

# instantiate the function do_stuff
my_func = my_class.do_stuff()

# I am manipulating a data array of size ~ 300 x 256 x 1 x 2
data = my_data  # my_data is imported just once and has the size above

# instantiate a 3d array of size 20 x 10 x 10 and fill it with all zeros
my_array = numpy.zeros([20,10,10]) 
# loop over parameters and fill array with desired output
for i in range(loop_1):
    for j in range(loop_2):
        for k in range(loop_3):
            # create tmp_data that has a shape which is the same as data except the first dimension can range from 1 - 1024 instead of being fixed at 300

            '''  Is the next line where I am causing memory problems? '''
            tmp_data = my_class.chop_data(data,i,j,k)
            my_func(tmp_data)
            my_func.third_party_function()
            my_array([i,j,k]) = my_func.results() # this is just a floating point number

            ''' should I do something to flush tmp_data? '''
#############################################

Thanks,

Andre


More information about the Tutor mailing list