[issue26516] Add PYTHONMALLOC env var and add support for malloc debug hooks in release mode

[STINNER Victor]

New submission from STINNER Victor:

Attached patch:

- Add PYTHONMALLOC env var which accepts 4 values:

  * pymalloc: use pymalloc for PyObject_Malloc(), malloc for PyMem_Malloc() and PyMem_RawMalloc()
  * pymalloc_debug: pymalloc + debug hooks
  * malloc: use malloc for PyObject_Malloc(), PyMem_Malloc() and PyMem_RawMalloc()
  * malloc_debug: malloc + debug hooks

- Add support for debug hooks in release mode
- Add unit test for debug hooks in test_capi.py
- Add unit on misuse of memory allocators in test_capi.py

PYTHONMALLOC is used even if -E command line option is used, I prefer to keep the code simple. PYTHONMALLOC must be checked before the first call to any Python memory allocator, so it must occur very earlier. Well, it's simply the first instruction of main()...

My main use case is to be able to use debug hooks to detect:

* misuse of python memory allocators like object allocated with PyObject_Malloc() and freed with PyMem_Free()
* buffer overflow
* buffer underlow

----------
files: pymem.patch
keywords: patch
messages: 261413
nosy: haypo, lemburg, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Add PYTHONMALLOC env var and add support for malloc debug hooks in release mode
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file42098/pymem.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26516>
_______________________________________