Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS

[Bartosz Golaszewski]

2018-07-24 12:09 GMT+02:00 Bartosz Golaszewski <brgl at bgdev.pl>:
> 2018-07-23 21:51 GMT+02:00 Thomas Jollans <tjol at tjol.eu>:
>> On 23/07/18 20:02, Bartosz Golaszewski wrote:
>>> Hi!
>>
>> Hey!
>>
>>> A user recently reported a memory leak in python bindings (C extension
>>> module) to a C library[1] I wrote. I've been trying to fix it since
>>> but so far without success. Since I'm probably dealing with a space
>>> leak rather than actual memory leak, valgrind didn't help much even
>>> when using malloc as allocator. I'm now trying to use
>>> PYTHONMALLOCSTATS but need some help on how to interpret the output
>>> emitted it's enabled.
>>
>> Oh dear.
>>
>>>
>>> [snip]
>>>
>>> The number of pools in arena 53 continuously grows. Its size column
>>> says: 432. I couldn't find any documentation on what it means but I
>>> assume it's an allocation of 432 bytes. [...]
>>
>> I had a quick look at the code (because what else does one do for fun);
>> I don't understand much, but what I can tell you is that
>>  (a) yes, that is an allocation size in bytes, and
>>  (b) as you can see, it uses intervals of 8. This means that pool 53
>>      is used for allocations of 424 < nbytes <= 432 bytes. Maybe your
>>      breakpoint needs tweaking.
>>  (c) Try breaking on _PyObject_Malloc or pymalloc_alloc. I think they're
>>      called by both PyMem_Malloc and PyObject_Malloc.
>>
>> int _PyObject_DebugMallocStats(FILE *out)
>>
>> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L2435
>>
>> static int pymalloc_alloc(void *ctx, void **ptr_p, size_t nbytes)
>>
>> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L1327
>>
>>
>> Have fun debugging!
>>
>> -- Thomas
>>
>>

[snip!]

>
> I don't see any other allocation of this size. Can this be some bug in
> the interpreter?
>
> Bart

Ok so this is strange: I can fix the leak if I explicitly call
PyObject_Free() on the leaking object which is created by "calling"
its type. Is this normal? Shouldn't Py_DECREF() be enough? The
relevant dealloc callback is called from Py_DECREF() but the object's
memory is not freed.

Bart