[New-bugs-announce] [issue31911] Use malloc_usable_size() is pymalloc for realloc

[STINNER Victor]

New submission from STINNER Victor <victor.stinner at gmail.com>:

Objects/obmalloc.c contains an interesting comment:

    if (!address_in_range(p, pool)) {
        /* pymalloc is not managing this block.

           If nbytes <= SMALL_REQUEST_THRESHOLD, it's tempting to try to take
           over this block.  However, if we do, we need to copy the valid data
           from the C-managed block to one of our blocks, and there's no
           portable way to know how much of the memory space starting at p is
           valid.

           As bug 1185883 pointed out the hard way, it's possible that the
           C-managed block is "at the end" of allocated VM space, so that a
           memory fault can occur if we try to copy nbytes bytes starting at p.
           Instead we punt: let C continue to manage this block. */
        return 0;
    }

See also bpo-1185883.

We don't have to guess, it's possible to get the size of a memory block allocated by malloc() with:

* malloc_usable_size(ptr): available at least on Linux
* _msize(ptr): Windows

Maybe we could add a "msize" field to PyMemAllocatorEx? See also bpo-18835 whic adds PyMem_AlignedAlloc() and so already modify (and rename PyMemAllocatorEx).

See also bpo-31626, but I'm not sure that it would help for _PyMem_DebugRawRealloc().

----------
components: Interpreter Core
messages: 305319
nosy: haypo, serhiy.storchaka, skrah
priority: normal
severity: normal
status: open
title: Use malloc_usable_size() is pymalloc for realloc
type: performance
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31911>
_______________________________________