Detecting memory leaks on apache, mod_python

[Fredrik Lundh]

Steven D'Aprano wrote:

>>  > Not me.
>>
>> You're quite knew to this internet thing, aren't you? ;-)
> 
> :-D

(hmm. why is that whenever you make some silly last-second addition to a 
post, you end up making a stupid typo?)

>> And things like "how much memory is free in the heap" isn't even a
>> meaningful concept on a modern machine, thanks to the wonders of virtual
>> memory (especially the overcommitting kind).
> 
> Maybe the memory model used in modern multi-tasking virtual-memory PCs 
> makes the concept of "free memory" obsolete. Although if so, somebody 
> should have a quiet word with the author of the Linux free command:
> 
> $ free
>              total       used       free     shared    buffers     cached
> Mem:       1002524     988736      13788          0       7044      98916
> -/+ buffers/cache:     882776     119748
> Swap:      4241080    3939736     301344
> 
> (Admittedly that's system-wide memory usage, rather than for a single 
> process.)

the problem isn't that you can get a number, it's that the number you 
get has no *direct* correlation to the amount of memory your process can 
actually allocate -- or at least allocate without causing excessive 
swapping or triggering the OOM killer or otherwise annoying all the 
other processes on the machine.

>> For Python, standard process monitoring tools (combined with a basic
>> understanding of how dynamic memory allocation works on modern
>> platforms) are usually sufficient to get a good view of an application's
>> memory usage patterns.  Just run the program under a few different
>> scenarios, and see what it does.
> 
> Are you saying that Python programs can't monitor their own memory use?
> 
> I'm happy to accept that "free memory" is not a meaningful concept for a 
> process in a modern system. That makes sense. But surely it is reasonable 
> for a process to have an idea of how much memory it has actually used. 
> Yes? No?

unless you do utterly trivial things, a process allocates memory from a 
lot of different resource pools (i/o buffers, virtual memory buffers, 
network buffers, graphics resources, etc).  there's simply no way for 
the process itself to know how much memory it uses.  if you want to 
know, ask the operating system.

>> If the memory use looks suspicious,
>> use standard debugging techniques to locate the problematic area, and
>> standard benchmarking techniques to look for unexpected blowups and
>> leaks.
> 
> What sort of "standard debugging techniques" work in the absence of any 
> way (that I know of) to measure memory usage?

as I said, use standard process monitoring tools (i.e. "ps" and "top" 
etc on Unix, the task manager on Windows) takes you a long way.

 > In Python, standard
> debugging techniques usually start with the print statement
 > but one  can't do anything like this:
> 
> # problematic area of code
> last = memory()
> for i in xrange(100):
>     x = foo()
>     if memory() >= last:
>         print "memory use increased", memory()

it doesn't have to be harder than this:

   # problematic area of code
   raw_input("before: check memory here")
   for i in xrange(100):
       x = foo()
   raw_input("after: check memory here")

on unix, you can also do e.g. os.system("ps ux") or os.system("ps -F -p 
%d" % os.getpid()) or some variation thereof.

> So what are you suggesting is standard?

running individual modules/classes/functions under test harnesses, so 
you can analyze their behaviour under well-known conditions.

</F>