[Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy, related.

[Bruce Southey]

On Sun, May 22, 2011 at 1:17 PM, Jeffrey Spencer <jeffspencerd at gmail.com> wrote:
> Further isolated the error and it seems to come in from the sum function in
> numpy. I will probe further later but the code below causes the memory usage
> to go up and up. If you change sum below to some other function there are no
> issues.
>
> from numpy import arange, sum
>
> for x in range(10000000):
>         inhibVal = sum(arange(15))
>
> Cheers,
> Jeff
>
> Message: 3
> Date: Sun, 22 May 2011 09:32:10 -0500
> From: Bruce Southey <bsouthey at gmail.com>
> Subject: Re: [Numpy-discussion] Python memory management issues using
> 	Linux. Maybe Numpy, related.
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID: <BANLkTinAuFp0o6OTwvP=GDhBKNkhkwH-vg at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Sun, May 22, 2011 at 8:42 AM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
>
>>
>>
>> On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer <jeffspencerd at gmail.com>
>> wrote:
>
>>>
>>> Sorry. I attached here the class and script to run it. It is just a
>>> minimalistic example of what I'm really doing that exhibits the same
>>> behavior. All the script does is run at.createSpecific after
>>> instantiating
>>> the class. I have seen this exhibited on many other cases but it runs
>>> fine
>>> in Windows XP. Here is what the script does:
>>>
>>> import MemoryLeak
>>> at = MemoryLeak.Model()
>>> at.createSpecific()
>>>
>
>> That manages to eat up all my cpu cycles for about 5 minutes, but no leak
>> (on OS X). I also don't see anything in the code that can potentially give
>> problems. I suggest you strip this down a lot more so you can isolate the
>> problem.
>>
>> If in your original code you were making a lot of figures in one of those
>> for loops without explicitly closing them, that could perhaps be a
>> problem.
>> But you removed all I/O and matplotlib related code already (except for
>> imports and some keywords).
>>
>> Ralf
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> A good OS is likely to keep Python module in memory just because it
> knows that it might be requested again. However, from the little I
> know (too much LWN), if Linux needs that memory it will reclaim that
> memory only when it needs it or part of some other process.
>
> Measuring memory usage using simple tools (like ps or top) are very
> far from sufficient, so exactly how do you know that the memory has
> not been released?
>
> Bruce
>
>
> --
> ________________________
> Jeffrey Spencer
> jeffspencerd at gmail.com
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
That example is just memory exhaustion not a memory leak because the
OS does reclaim that memory. Sure it is not immediate but it does
eventually happen otherwise you would end up to no memory left. Please
note that numpy requires contiguous memory so allocating huge arrays,
relative to the amount of available memory, in such a manner *will*
cause thrashing regardless.

Also take a look at Pythons' garbage collection module
(http://docs.python.org/library/gc.html) and perhaps use manual
garbage collection. There is  a lot out there on this.

As per Wikipedia
(http://en.wikipedia.org/wiki/Memory_leak#Other_memory_consumers)
"Note that constantly increasing memory usage is not necessarily
evidence of a memory leak." But probably more relevant definitions
are:
http://stackoverflow.com/questions/312069/the-best-memory-leak-definition

Bruce