[Tutor] Memory Problem Continued
DORSEY_EDMUND_K@LILLY.COM
DORSEY_EDMUND_K@LILLY.COM
Mon Jul 7 12:40:03 2003
This is a multipart message in MIME format.
--=_alternative 005B759405256D5C_=
Content-Type: text/plain; charset="us-ascii"
Previously I was having a memory problem and it was pointed out that the
range function is quite the memory hog and to use xrange instead.
Unfortunately this didn't solve the problem (though it did fix a problem
that I would have probably encountered later with larger data sets) I
don't like just pasting code into my email but I'm afraid it's the only
way I can explain the problem. So here it goes. When I start up the
program and it loads the data set it uses up a fixed amount of memory. As
the subvoxelize function shown below continues to run (takes about 5
minutes before crashing) the memory usage just keeps increasing at a
constant rate until I run out of memory.
def subvoxelize(self, progressFunc=None):
value = 0
updateNum = self._numVoxelsOrig/100
for n in xrange(self._numVoxelsOrig): #self._numVoxelsOrig has a
value of around 20 million
x, y, z = self.to3d(n) #not
causing any problems with memory
x2, y2, z2 = x*2, y*2, z*2
index0 = x2 + y2*self._newXsize + z2 * (self._newXY)
#here is where the problem is and with the subsequent assignments
index1 = x2+1 + y2*self._newXsize + z2 * (self._newXY)
index2 = x2 + (y2+1)*self._newXsize + z2 * (self._newXY)
index3 = x2+1 + (y2+1)*self._newXsize + z2 * (self._newXY)
index4 = x2 + y2*self._newXsize + (z2+1) *
(self._newXY)
index5 = x2+1 + y2*self._newXsize + (z2+1) * (self._newXY)
index6 = x2 + (y2+1)*self._newXsize + (z2+1) *
(self._newXY)
index7 = x2+1 + (y2+1)*self._newXsize + (z2+1) * (self._newXY)
#if I don't do these assignments and make each index go to a constant it
doesn't leak memory
self._newData[index0] = self._data[n]
self._newData[index1] = self._data[n]
self._newData[index2] = self._data[n]
self._newData[index3] = self._data[n]
self._newData[index4] = self._data[n]
self._newData[index5] = self._data[n]
self._newData[index6] = self._data[n]
self._newData[index7] = self._data[n]
if n % updateNum == 0:
progressFunc(value)
value = value+1
return self._newData
That is my function. I have narrowed the problem down to where I assign
values to index 0 through 7.
If I just assign those values to some constant ie. 0 it doesn't eat up my
memory. (shown below)
index0 = 0
index1 =0
index2 = 0 .... #doing this works
Does anyone have any idea why I would be "leaking" memory?? The above
algorithm works fine with smaller data sets and even does what I want it
to do but anything large and it uses up all the memory and dies. Thank
you for any advice. ~Ed
--=_alternative 005B759405256D5C_=
Content-Type: text/html; charset="us-ascii"
<br><font size=2 face="sans-serif">Previously I was having a memory problem and it was pointed out that the range function is quite the memory hog and to use xrange instead. Unfortunately this didn't solve the problem (though it did fix a problem that I would have probably encountered later with larger data sets) I don't like just pasting code into my email but I'm afraid it's the only way I can explain the problem. So here it goes. When I start up the program and it loads the data set it uses up a fixed amount of memory. As the subvoxelize function shown below continues to run (takes about 5 minutes before crashing) the memory usage just keeps increasing at a constant rate until I run out of memory. </font>
<br>
<br><font size=2 face="sans-serif"> def subvoxelize(self, progressFunc=None):</font>
<br><font size=2 face="sans-serif"> value = 0</font>
<br><font size=2 face="sans-serif"> updateNum = self._numVoxelsOrig/100</font>
<br><font size=2 face="sans-serif"> for n in xrange(self._numVoxelsOrig): #self._numVoxelsOrig has a value of around 20 million</font>
<br><font size=2 face="sans-serif"> x, y, z = self.to3d(n) #not causing any problems with memory</font>
<br><font size=2 face="sans-serif"> x2, y2, z2 = x*2, y*2, z*2 </font>
<br><font size=2 face="sans-serif"> index0 = x2 + y2*self._newXsize + z2 * (self._newXY) #here is where the problem is and with the subsequent assignments</font>
<br><font size=2 face="sans-serif"> index1 = x2+1 + y2*self._newXsize + z2 * (self._newXY)</font>
<br><font size=2 face="sans-serif"> index2 = x2 + (y2+1)*self._newXsize + z2 * (self._newXY)</font>
<br><font size=2 face="sans-serif"> index3 = x2+1 + (y2+1)*self._newXsize + z2 * (self._newXY)</font>
<br><font size=2 face="sans-serif"> index4 = x2 + y2*self._newXsize + (z2+1) * (self._newXY)</font>
<br><font size=2 face="sans-serif"> index5 = x2+1 + y2*self._newXsize + (z2+1) * (self._newXY)</font>
<br><font size=2 face="sans-serif"> index6 = x2 + (y2+1)*self._newXsize + (z2+1) * (self._newXY)</font>
<br><font size=2 face="sans-serif"> index7 = x2+1 + (y2+1)*self._newXsize + (z2+1) * (self._newXY) #if I don't do these assignments and make each index go to a constant it doesn't leak memory</font>
<br><font size=2 face="sans-serif"> self._newData[index0] = self._data[n]</font>
<br><font size=2 face="sans-serif"> self._newData[index1] = self._data[n]</font>
<br><font size=2 face="sans-serif"> self._newData[index2] = self._data[n]</font>
<br><font size=2 face="sans-serif"> self._newData[index3] = self._data[n]</font>
<br><font size=2 face="sans-serif"> self._newData[index4] = self._data[n]</font>
<br><font size=2 face="sans-serif"> self._newData[index5] = self._data[n]</font>
<br><font size=2 face="sans-serif"> self._newData[index6] = self._data[n]</font>
<br><font size=2 face="sans-serif"> self._newData[index7] = self._data[n]</font>
<br><font size=2 face="sans-serif"> if n % updateNum == 0:</font>
<br><font size=2 face="sans-serif"> progressFunc(value)</font>
<br><font size=2 face="sans-serif"> value = value+1</font>
<br><font size=2 face="sans-serif"> return self._newData</font>
<br>
<br>
<br><font size=2 face="sans-serif">That is my function. I have narrowed the problem down to where I assign values to index 0 through 7. </font>
<br><font size=2 face="sans-serif">If I just assign those values to some constant ie. 0 it doesn't eat up my memory. (shown below)</font>
<br>
<br><font size=2 face="sans-serif">index0 = 0</font>
<br><font size=2 face="sans-serif">index1 =0</font>
<br><font size=2 face="sans-serif">index2 = 0 .... #doing this works</font>
<br>
<br><font size=2 face="sans-serif">Does anyone have any idea why I would be "leaking" memory?? The above algorithm works fine with smaller data sets and even does what I want it to do but anything large and it uses up all the memory and dies. Thank you for any advice. ~Ed</font>
<br>
<br>
--=_alternative 005B759405256D5C_=--