[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Brett Cannon]

On Wed, 9 Mar 2016 at 06:57 Victor Stinner <victor.stinner at gmail.com> wrote:

> 2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> >> Perhaps if you add some guards somewhere :-)
> >
> > We have runtime checks but only implemented in debug mode for efficiency.
> >
> > By the way, I proposed once to add an environment variable to allow to
> > enable these checks without having to recompile Python.  Since the PEP
> > 445, it became easy to implement this. What do you think?
> >
> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable
>
> Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable:
>
>    http://bugs.python.org/issue26516
>
> PYTHONMALLOC=debug installs debug hooks to:
>
> * detect API violations, ex: PyObject_Free() called on a buffer
> allocated by PyMem_Malloc()
> * detect write before the start of the buffer (buffer underflow)
> * detect write after the end of the buffer (buffer overflow)
>
> https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks
>
> The main advantage of this variable is that you don't have to
> recompile Python in debug mode to benefit of these checks.
>

I just wanted to say this all sounds awesome! Thanks for all the hard work
on making our memory management story easier to work with, Victor.

-Brett


>
> Recompiling Python in debug mode requires to recompile *all*
> extensions modules since the debug ABI is incompatible. When I played
> with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/
> ), I had such issues, it was very annoying with non-trivial extension
> modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to
> recompile extension modules anymore!
>
>
> With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool
> suite to "debug memory"!
>
> My motivation for PYTHONMALLOC=debug is to detect API violations to
> prepare my change on PyMem_Malloc() allocator (
> http://bugs.python.org/issue26249 ), but also to help users to detect
> bugs.
>
> It's common that users report a bug: "Python crashed", but have no
> idea of the responsible of the crash. I hope that detection of buffer
> underflow & overflow will help them to detect bugs in their own
> extension modules.
>
>
> Moreover, I added PYTHONMALLOC=malloc to ease the use of external
> memory debugger on Python. By default, Python uses pymalloc allocator
> for PyObject_Malloc() which raises a lot of false positive in
> Valgrind. We even have a configuration (--with-valgrind) and a
> Valgrind suppressino file to be able to skip these false alarms in
> Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind
> (or other tools).
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160309/61eb5cdf/attachment.html>