[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

Ping? Is someone still opposed to my change #26249 "Change
PyMem_Malloc to use pymalloc allocator"? If no, I think that I will
push my change.

My change only changes two lines, so it can be easily reverted before
CPython 3.6 if we detect major issues in third-party extensions. And
maybe it's better to push such change today to get more time to play
with it, than pushing it late in the development of CPython 3.6.

The new PYTHONMALLOC=debug feature allows to quickly and easily check
the usage of the PyMem_Malloc() API, even if Python is compiled in
release mode.

I checked multiple Python extensions written in C. I only found one
bug in numpy and I sent a patch (not merged yet).

victor

2016-03-15 0:19 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> 2016-02-12 14:31 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>>>> If your program has bugs, you can use a debug build of Python 3.5 to
>>>> detect misusage of the API.
>>
>> Yes, but people don't necessarily do this, e.g. I have
>> for a very long time ignored debug builds completely
>> and when I started to try them, I found that some of the
>> things I had been doing with e.g. free list implementations
>> did not work in debug builds.
>
> I just added support for debug hooks on Python memory allocators on
> Python compiled in *release* mode. Set the environment variable
> PYTHONMALLOC to debug to try with Python 3.6.
>
> I added a check on PyObject_Malloc() debug hook to ensure that the
> function is called with the GIL held. I opened an issue to add a
> similar check on PyMem_Malloc():
> https://bugs.python.org/issue26563
>
>
>> Yes, but those are part of the stdlib. You'd need to check
>> a few C extensions which are not tested as part of the stdlib,
>> e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom
>> types in C since these will often need the memory management
>> APIs).
>>
>> It may also be a good idea to check wrapper generators such
>> as cython, swig, cffi, etc.
>
> I ran the test suite of numpy, lxml, Pillow and cryptography (used cffi).
>
> I found a bug in numpy. numpy calls PyMem_Malloc() without holding the GIL:
> https://github.com/numpy/numpy/pull/7404
>
> Except of this bug, all other tests pass with PyMem_Malloc() using
> pymalloc and all debug checks.
>
> Victor