Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS

[Bartosz Golaszewski]

Hi!

A user recently reported a memory leak in python bindings (C extension
module) to a C library[1] I wrote. I've been trying to fix it since
but so far without success. Since I'm probably dealing with a space
leak rather than actual memory leak, valgrind didn't help much even
when using malloc as allocator. I'm now trying to use
PYTHONMALLOCSTATS but need some help on how to interpret the output
emitted it's enabled.

I'm setting PYTHONMALLOCSTATS=1 & PYTHONMALLOC=pymalloc_debug and then
running the script that triggers the leak. The last debug message is
as follows:

class   size   num pools   blocks in use  avail blocks
-----   ----   ---------   -------------  ------------
    3     32           1               2           124
    4     40           2               9           193
    5     48           1               3            81
    6     56           1               4            68
    7     64          11             295           398
    8     72           9             260           244
    9     80          36             831           969
   10     88          79            1542          2092
   11     96         131            3262          2240
   12    104          70            1903           757
   13    112          19             289           395
   14    120          11             139           224
   15    128           7              88           129
   16    136           6              70           104
   17    144           5              44            96
   18    152           4              47            57
   19    160          24             342           258
   20    168           4              17            79
   21    176          24             360           192
   22    184           2               8            36
   23    192           2              11            31
   24    200          22             227           213
   25    208           3              13            44
   26    216           3               7            47
   27    224           2              13            23
   28    232           2               6            28
   29    240           3               8            40
   30    248           2              10            22
   31    256           3              10            35
   32    264           2               9            21
   33    272           3              11            31
   34    280           2              10            18
   35    288           1               3            11
   36    296           2               9            17
   37    304           2               9            17
   38    312           2               5            19
   39    320           2               5            19
   40    328          14             105            63
   41    336           2               3            21
   42    344           1               3             8
   43    352           1               3             8
   44    360           2               3            19
   45    368           1               3             8
   46    376           1               1             9
   47    384           2               4            16
   48    392           2               6            14
   49    400           2               3            17
   50    408           1               1             8
   51    416           1               3             6
   52    424           2               4            14
   53    432       50967          458680            23
   54    440           3               9            18
   55    448           4              15            21
   56    456           4              12            20
   57    464           3               8            16
   58    472           2               5            11
   59    480           1               4             4
   60    488           1               3             5
   61    496           4              11            21
   62    504           4              13            19
   63    512           2               7             7

# times object malloc called       =            2,811,245
# arenas allocated total           =                  810
# arenas reclaimed                 =                    0
# arenas highwater mark            =                  810
# arenas allocated current         =                  810
810 arenas * 262144 bytes/arena    =          212,336,640

# bytes in allocated blocks        =          199,277,432
# bytes in available blocks        =            1,138,472
308 unused pools * 4096 bytes      =            1,261,568
# bytes lost to pool headers       =            2,473,536
# bytes lost to quantization       =            8,185,632
# bytes lost to arena alignment    =                    0
Total                              =          212,336,640

The number of pools in arena 53 continuously grows. Its size column
says: 432. I couldn't find any documentation on what it means but I
assume it's an allocation of 432 bytes. I launched gdb and set the
following breakpoint: 'b PyMem_Malloc if size == 432' but the
breakpoint was never triggered. I tried the same for PyObject_Malloc
and still nothing.

How do I use the info produced by PYTHONMALLOCSTATS do get to the
culprit of the leak? Is there anything wrong in my reasoning here?

Best regards,
Bartosz Golaszewski

[1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/