[Python-Dev] Re: Activating pymalloc

[Tim Peters]

[The elusive Vladimir Marangozov]
> I am trying to follow remotely python-dev from time to time
> but I can't be very active as I used to be. Sorry about that.

That's OK -- we all get old and useless at our own private pace <wink>.

> ...
> 1. The docs are not perfect and should be updated. Especially
>    what Tim mentioned -- adding a table summarizing the APIs
>    would be helpful. My original intention for the docs was to
>    strip off some internal details that the usual extension
>    writer doesn't have to know about. But Guido is right that
>    the comments in the .h files are the reference and they are
>    probably much clearer.

They were better at the start, too:  all the explanatory text wound up
getting spread out over a seemingly random collection of .h and .c files,
and there's no single "memory mgmt" section in the manuals to tie it all
together either.  That all seems fixable, though.

> 2. I agree that the macro chain (especially on the pymalloc side)
>    is not so useful at the end, so maybe all PyCore_ macros can be
>    removed. The function names of the allocator can be cast in
>    stone and then _THIS_xxx in obmalloc.c replaced with them.

Have you looked at Neil's patch?

> 3. I do not agree however that one would want to explicitly call
>    pymalloc. Access to the object allocator should be done through
>    PyObject_xxx, and to the memory allocator through PyMem_.

I'm not sure whether this is a real disagreement, or about what words mean.
There are some cases in the core now where we probably should be using
obmalloc.c for "raw" memory.  PyMem_XXX continues to refer to platform
malloc etc (unless an adventurous soul redefineds them); while PyObject_XXX
seem *conceptually* unsuited to "raw" memory.  Maybe that's an error in my
conception, though <wink>.

> 4. Originally, I excluded PyMem_ to use pymalloc because profiling
>    showed that more than 90% of all small allocations are object
>    allocations. So non-object allocations were either big (in which
>    case pymalloc is just overhead), either accounted for a very
>    small percentage which can be neglected. This is the reason why
>    pymalloc became obmalloc. Practical reasons.

Understood, and all that is probably still qualitatively valid (the specific
cutoff points have probably shifted, since various popular Python objects
are larger now).  The most frightening thing to me about ever switching
PyMem_XXX away from the platform allocator are thread issues:  I want to
retain obmalloc's expoitation of the GIL, but people currently have no fear
of using PyMem_XXX when not holding the GIL.

> 5. About breaking extensions -- in early versions of pymalloc I had
>    debugging support built-in which basically detected when a block
>    allocated with malloc was free'd or realloc'd with pymalloc.
>    I used this at the time to cleanup the baseline and the standard
>    extensions shipped with 2.0 from mixed API usage. After that was
>    done, though, I removed this as I thought that it won't be of
>    much help. Apparently, I was wrong and perhaps this debugging
>    functionality would be helpful. Still, the way to go is to fix
>    the 'faulty' modules and document better the concepts. Yes, 1.5.2
>    would be the primary victims here but the memory work was done
>    post 1.5.2 and people can't stop Python from evolving.

Heh.  They've sure tried <wink>.

> ...
> More generally, as a historical recap, the goal originally was to
> introduce the core concepts on Python heap & Python object heap
> through the APIs which was a precondition for any python-specific
> allocator work. Once this was done, the integration of a python
> malloc had to be as seemless as possible, the switching to
> its use as well, and this is probably where some of the macro
> layers were introabused (in my eyes, not a big deal anyway). So look
> at it on the bright side -- the goals are now almost achieved :-)

Please don't take anything here as denigrating your work!  You left this all
in much better shape than you found it, and it's appreciated.  BTW, over the
last year we've seen several real-life Python programs that managed to
provoke Linux flavors into quadratic-time free() behavior.  In all such
cases, using pymalloc made the problem go away, and provided enormous
speedups.  Lots of people have reported modest speedups in non-pathological
programs too (except on OS/2 EMX, or something like that).

> Anything else I can help you with?

Well, there are several hundred open bugs and patches on SourceForge.  If
you don't want to figure out how threads *really* work on HP-UX, I guess you
can do for us what all the semi-retired do:  let us blame you for everything
someone doesn't like <wink>.

just-leave-your-kidneys-at-the-door-on-your-way-out-ly y'rs  - tim