Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS

[Thomas Jollans]

On 23/07/18 20:02, Bartosz Golaszewski wrote:
> Hi!

Hey!

> A user recently reported a memory leak in python bindings (C extension
> module) to a C library[1] I wrote. I've been trying to fix it since
> but so far without success. Since I'm probably dealing with a space
> leak rather than actual memory leak, valgrind didn't help much even
> when using malloc as allocator. I'm now trying to use
> PYTHONMALLOCSTATS but need some help on how to interpret the output
> emitted it's enabled.

Oh dear.

> 
> [snip]
> 
> The number of pools in arena 53 continuously grows. Its size column
> says: 432. I couldn't find any documentation on what it means but I
> assume it's an allocation of 432 bytes. [...]

I had a quick look at the code (because what else does one do for fun);
I don't understand much, but what I can tell you is that
 (a) yes, that is an allocation size in bytes, and
 (b) as you can see, it uses intervals of 8. This means that pool 53
     is used for allocations of 424 < nbytes <= 432 bytes. Maybe your
     breakpoint needs tweaking.
 (c) Try breaking on _PyObject_Malloc or pymalloc_alloc. I think they're
     called by both PyMem_Malloc and PyObject_Malloc.

int _PyObject_DebugMallocStats(FILE *out)

https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L2435

static int pymalloc_alloc(void *ctx, void **ptr_p, size_t nbytes)

https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L1327


Have fun debugging!

-- Thomas


> 
> How do I use the info produced by PYTHONMALLOCSTATS do get to the
> culprit of the leak? Is there anything wrong in my reasoning here?
> 
> Best regards,
> Bartosz Golaszewski
> 
> [1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/
>