Memory Problem

[Christoph Scheit]

Hi,

I have a short script/prog in order to read out binary files from a numerical 
simulation. This binary files still need some post-processing, which is
summing up results from different cpu's, filtering out non-valid entrys
and bringing the data in some special order.

Reading the binary data in using the struct-module works fine - I read
one chunk of data into a tuple, this tupel I append to a list.
At the end of reading, I return the list.

Then the data is added to a table, which I use for the actual Post-Processing.
The table is actually a Class with several "Columns", each column internally
being represented by array.
Now adding all the data from the simulation results to the table makes the
memory usage exploding. So I would like to know, where exactly the memory
is vasted.

Here the code to add the data of one file (I have to add the data of various 
files to the same table in total)

# create reader
breader = BDBReader("<var>", "<type>", "#")
 
# read data
bData = breader.readDB(dbFileList[0])

# create table
dTab = DBTable(breader.headings, breader.converters, [1,2])
addRows(bData, dTab)

Before I add a new entry to the table, I check if there is already an entry 
like this. To do so, I store keys for all the entries with row-number in a
dictionary. What about the memory consumption of the dictionary?

Here the code for adding a new row to the table:

# check if data already exists
if (self.keyDict.has_key(key)):
        rowIdx = self.keyDict[key]
        for i in self.mutableCols:
            self.cols[i][rowIdx] += rowData[i]
        return
            
 # key is still available - insert row to table
 self.keyDict[key] = self.nRows
        
 # insert data to the columns
 for i in range(0, self.nCols):
     self.cols[i].add(rowData[i])
        
 # add row i and increment number of rows
 self.rows.append(DBRow(self, self.nRows))
 self.nRows += 1

Maybe somebody can help me. If you need, I can give more implementation
details.

Thanks in advance,

Christoph
-- 

============================
M.Sc. Christoph Scheit
Institute of Fluid Mechanics
FAU Erlangen-Nuremberg
Cauerstrasse 4
D-91058 Erlangen
Phone: +49 9131 85 29508
============================