[Python-Dev] tracemalloc: add an optional memory limit

[Antoine Pitrou]

On Mon, 9 Dec 2013 10:28:17 +0100
Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
> 
> 2013/12/9 Serhiy Storchaka <storchaka at gmail.com>:
> > But tracemalloc doesn't count memory allocated besides Python allocators
> > (e.g. memory for executable, static variables and stack, memory allocated by
> > extensions and C lib, unallocated but not returned to OS dynamical memory).
> > When you want investigate how your program works on low memory, e.g. 500 MB,
> > ulimit -v 500000 is much closer to real life than tracemalloc.
> 
> Well, both limits are useful, but also (completly?) different.
> 
> "memory for executable, static variables and stack," : these values
> are fixed, so they don't matter at all. Substract them from the RSS
> when you choose the memory limit for tracemalloc.

Stack consumption is O(number of threads).

> "unallocated but not returned to OS dynamical memory" : it's really
> hard to estimate and understand the fragmentation of the heap memory
> :-( But this may only concern short peak, and the free memory can be
> reused later. If the peak is still lower than the limit, the
> fragmentation does not matter

I don't understand the statement: "if the peak is still lower than the
limit, the fragmentation does not matter". Serhiy's point is precisely
that the peak's computation doesn't account for OS-level fragmentation.

That said, I agree the feature can be useful. Unlike ulimit, it's a
best-effort thing, but it also works at a much more useful level than
"size of the virtual address space".

Regards

Antoine.