[issue3329] API for setting the memory allocator used by Python

[STINNER Victor]

STINNER Victor added the comment:

py_setallocators-filename.patch: Here is a try to define an API providing the filename and line number of the C code. The Py_SetAllocators() API is unchanged:

PyAPI_FUNC(int) Py_SetAllocators(
    char api,
    void* (*malloc) (size_t size, void *user_data),
    void* (*realloc) (void *ptr, size_t size, void *user_data),
    void (*free) (void *ptr, void *user_data),
    void *user_data
    );

If Python is compiled with -DPYMEM_TRACE_MALLOC, user_data is not the last parameter passed to Py_SetAllocators() but a pointer to a _PyMem_Trace structure:

typedef struct {
    void *data;
    /* NULL and -1 when unknown */
    const char *filename;
    int lineno;
} _PyMem_Trace;


The problem is that the module using Py_SetAllocators() must be compiled differently depending on PYMEM_TRACE_MALLOC. Example from pytracemalloc, modified for this patch:
---
    _PyMem_Trace *ctrace;
    trace_api_t *api;
    void *call_data;
    void *ptr;
#ifdef PYMEM_TRACE_MALLOC
    ctrace = (_PyMem_Trace *)data;
    api = (trace_api_t *)ctrace->data;
    ctrace->data = api->data;
    call_data = data;
#else
    ctrace = NULL;
    api = (trace_api_t *)data;
    call_data = api->data;
#endif
    ptr = api->malloc(size, call_data);
    ...
---
I didn't like the "ctrace->data = api->data;" instruction: pytracemalloc modifies the input _PyMem_Trace structure.


pytracemalloc code is a little bit more complex, but "it works". pytracemalloc can reuse the filename and line number of the C module, or of the Python module. It can be configured at runtime. Example of output for the C module:
---
2013-06-11 00:36:30: Top 15 allocations per file and line (compared to 2013-06-11 00:36:25)
#1: Objects/dictobject.c:352: size=6 MiB (+4324 KiB), count=9818 (+7773), average=663 B
#2: Objects/unicodeobject.c:1085: size=6 MiB (+2987 KiB), count=61788 (+26197), average=111 B
#3: Objects/tupleobject.c:104: size=4054 KiB (+2176 KiB), count=44569 (+24316), average=93 B
#4: Objects/typeobject.c:770: size=2440 KiB (+1626 KiB), count=13906 (+10360), average=179 B
#5: Objects/bytesobject.c:107: size=2395 KiB (+1114 KiB), count=24846 (+11462), average=98 B
#6: Objects/funcobject.c:12: size=1709 KiB (+1103 KiB), count=11516 (+7431), average=152 B
#7: Objects/codeobject.c:117: size=1760 KiB (+871 KiB), count=11267 (+5578), average=160 B
#8: Objects/dictobject.c:399: size=784 KiB (+627 KiB), count=10040 (+8028), average=80 B
#9: Objects/listobject.c:159: size=420 KiB (+382 KiB), count=5386 (+4891), average=80 B
#10: Objects/frameobject.c:649: size=1705 KiB (+257 KiB), count=3374 (+505), average=517 B
#11: ???:?: size=388 KiB (+161 KiB), count=588 (+240), average=676 B
#12: Objects/weakrefobject.c:36: size=241 KiB (+138 KiB), count=2579 (+1482), average=96 B
#13: Objects/dictobject.c:420: size=135 KiB (+112 KiB), count=2031 (+1736), average=68 B
#14: Objects/classobject.c:59: size=109 KiB (+105 KiB), count=1400 (+1345), average=80 B
#15: Objects/unicodeobject.c:727: size=188 KiB (+86 KiB), count=1237 (+687), average=156 B
37 more: size=828 KiB (+315 KiB), count=8421 (+5281), average=100 B
Total Python memory: size=29 MiB (+16 MiB), count=212766 (+117312), average=145 B
Total process memory: size=68 MiB (+22 MiB) (ignore tracemalloc: 0 B)
---


I also had to modify the following GC functions to get more accurate information:

- _PyObject_GC_Malloc(size)
- _PyObject_GC_New(tp)
- _PyObject_GC_NewVar(tp, nitems)
- PyObject_GC_Del(op)

For example, PyTuple_New() calls PyObject_GC_NewVar() to allocate its memory. With my patch, you get "Objects/tupleobject.c:104" instead of a generic "Modules/gcmodule.c:1717".

----------
Added file: http://bugs.python.org/file30537/py_setallocators-filename.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3329>
_______________________________________