[Python-Dev] pymalloc and overallocation (unicodeobject.c,2.139,2.140 checkin)

[Martin v. Loewis]

Tim Peters <tim.one@comcast.net> writes:

> > Right. I confused this with the fact that PyMem_Realloc won't return
> > the excess memory,
> 
> PyMem_Realloc does whatever the system realloc does -- PyMem_Realloc doesn't
> go thru pymalloc today (except in a PYMALLOC_DEBUG build).  Doesn't matter,
> though, since strings use the PyObject_{Malloc, Free, Realloc} family today,
> and that does use pymalloc.  

That's what I mean (I'm *really* confused about memory family APIs,
ever since everything changed :-)

> OTOH, there's no reason PyObject_Realloc *has* to hang on to all
> small-block memory on a shrinking realloc, and there's no reason
> pymalloc couldn't grow another realloc entry point specifying what
> the caller wants a shrinking realloc to do.  These things are all
> easy to change, but I don't know what's truly desirable.

Neither do I. To establish whether releasing the extra memory is worth
the effort would depend on knowledge whether the object will be
long-living; neither pymalloc nor its caller is able to tell.

> Note another subtlety:  I expect you brought up PyMem_Realloc because
> unicodeobject.c uses the PyMem_XYZ family for managing the
> PyUnicodeObject.str member today.  

No, because I assumed PyMem_Realloc was a synonym for
PyObject_Realloc.

> That means it normally never uses pymalloc at all, except to
> allocate fixed-size PyUnicodeObject structs (which use the
> PyObject_XYZ memory family).  I don't know whether that's the best
> idea, but that's how it is today.

I do think that the Unicode data should be managed by pymalloc as
well. Of course, DecodeUTF8 would then raise the same issue: decoding
UTF-8 doesn't know how many characters you'll get, either. This
currently does not try to be clever, but allocates enough memory for
the worst case.

Regards,
Martin