Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS

[Bartosz Golaszewski]

2018-07-24 13:30 GMT+02:00 Bartosz Golaszewski <brgl at bgdev.pl>:
> 2018-07-24 12:09 GMT+02:00 Bartosz Golaszewski <brgl at bgdev.pl>:
>> 2018-07-23 21:51 GMT+02:00 Thomas Jollans <tjol at tjol.eu>:
>>> On 23/07/18 20:02, Bartosz Golaszewski wrote:
>>>> Hi!
>>>
>>> Hey!
>>>
>>>> A user recently reported a memory leak in python bindings (C extension
>>>> module) to a C library[1] I wrote. I've been trying to fix it since
>>>> but so far without success. Since I'm probably dealing with a space
>>>> leak rather than actual memory leak, valgrind didn't help much even
>>>> when using malloc as allocator. I'm now trying to use
>>>> PYTHONMALLOCSTATS but need some help on how to interpret the output
>>>> emitted it's enabled.
>>>
>>> Oh dear.
>>>
>>>>
>>>> [snip]
>>>>
>>>> The number of pools in arena 53 continuously grows. Its size column
>>>> says: 432. I couldn't find any documentation on what it means but I
>>>> assume it's an allocation of 432 bytes. [...]
>>>
>>> I had a quick look at the code (because what else does one do for fun);
>>> I don't understand much, but what I can tell you is that
>>>  (a) yes, that is an allocation size in bytes, and
>>>  (b) as you can see, it uses intervals of 8. This means that pool 53
>>>      is used for allocations of 424 < nbytes <= 432 bytes. Maybe your
>>>      breakpoint needs tweaking.
>>>  (c) Try breaking on _PyObject_Malloc or pymalloc_alloc. I think they're
>>>      called by both PyMem_Malloc and PyObject_Malloc.
>>>
>>> int _PyObject_DebugMallocStats(FILE *out)
>>>
>>> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L2435
>>>
>>> static int pymalloc_alloc(void *ctx, void **ptr_p, size_t nbytes)
>>>
>>> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L1327
>>>
>>>
>>> Have fun debugging!
>>>
>>> -- Thomas
>>>
>>>
>
> [snip!]
>
>>
>> I don't see any other allocation of this size. Can this be some bug in
>> the interpreter?
>>
>> Bart
>
> Ok so this is strange: I can fix the leak if I explicitly call
> PyObject_Free() on the leaking object which is created by "calling"
> its type. Is this normal? Shouldn't Py_DECREF() be enough? The
> relevant dealloc callback is called from Py_DECREF() but the object's
> memory is not freed.
>
> Bart

Ok I've found the problem and it's my fault. From tp_dealloc's documentation:

---
The destructor function should free all references which the instance
owns, free all memory buffers owned by the instance (using the freeing
function corresponding to the allocation function used to allocate the
buffer), and finally (as its last action) call the type’s tp_free
function.
---

I'm not calling the tp_free function...

Best regards,
Bartosz Golaszewski