[Python-Dev] cpython: Implement PEP 412: Key-sharing dictionaries (closes #13903)

[Kristján Valur Jónsson]

> -----Original Message-----
> Take a look at the benchmark suite at
> http://hg.python.org/benchmarks/
> The test runner has an -m option that profiles memory usage, you could take
> a look at how that is implemented
>

Yes, out of process monitoring of memory as reported by the OS.  We do gather those counters as well on clients and servers.
But they don't give you the granularity you want when checking for memory leaks and memory usage by certain algorithms.
In the same way that the unittests have reference leak reports, they could just have memory usage reports, if the underlying allocator supported that.

FYI the current state of affairs of the cPython 2.7 branch we use is as follows:
1) We allow the API user to specify the base allocator python uses, both for regular allocs and allocating blocks for the obmalloc one, using:

/* Support for custom allocators */
typedef void *(*PyCCP_Malloc_t)(size_t size, void *arg, const char *file, int line, const char *msg);
typedef void *(*PyCCP_Realloc_t)(void *ptr, size_t size, void *arg, const char *file, int line, const char *msg);
typedef void (*PyCCP_Free_t)(void *ptr, void *arg, const char *file, int line, const char *msg);
typedef size_t (*PyCCP_Msize_t)(void *ptr, void *arg);
typedef struct PyCCP_CustomAllocator_t
{
    PyCCP_Malloc_t  pMalloc;
    PyCCP_Realloc_t pRealloc;
    PyCCP_Free_t    pFree;
    PyCCP_Msize_t   pMsize;    /* can be NULL, or return -1 if no size info is avail. */
    void            *arg;      /* opaque argument for the functions */
} PyCCP_CustomAllocator_t;

/* To set an allocator!  use 0 for the regular allocator, 1 for the block allocator.
 * pass a null pointer to reset to internal default
 */
PyAPI_FUNC(void) PyCCP_SetAllocator(int which, const PyCCP_CustomAllocator_t *); /* for BLUE to set the current context */

/* internal data member */
extern PyCCP_CustomAllocator_t _PyCCP_CustomAllocator[];

2) using ifdefs, the macros will delegate all final allocations through these allocators.  This includes all the "naked" malloc calls scattered about, they are patched up using #defines.

3) Additionally, there is an internal layer of management, before delegating to the external allocators.  This internal manager provides statistics, exposed through the "sys" module.

The layering is something like this, all more or less definable by pre-processor macros. (raw malloc() is turned into something else via pre-processor magic and a special "patch_malloc.h" file added to the modules which uses raw malloc())

          PyMem_Malloc()                         PyObject_Malloc()
                |                                           |
                v                                           v
           Mem bookkeeping                           obj bookkeeping
                |                                           |
                |                                           v
 malloc()       |                                     obmallocator
    |           |                                           |
    v           v                                           v
  PyMem_MALLOC_RAW()                             PyObject_MALLOC_RAW
           |                                       |
           v                                       v
     malloc() or vectored allocator specified through API function


Cheers,

K

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120425/680865b0/attachment.html>