[Python-Dev] Memory Allocator Part 2: Did I get it right?

[Tim Peters]

[Evan Jones]
> After I finally understood what thread-safety guarantees the Python
> memory allocator needs to provide, I went and did some hard thinking
> about the code this afternoon. I believe that my modifications provide
> the same guarantees that the original version did. I do need to declare
> the arenas array to be volatile, and leak the array when resizing it.
> Please correct me if I am wrong, but the situation that needs to be
> supported is this:

As I said before, I don't think we need to support this any more. 
More, I think we should not -- the support code is excruciatingly
subtle, it wasted plenty of your time trying to keep it working, and
if we keep it in it's going to continue to waste time over the coming
years (for example, in the short term, it will waste my time reviewing
it).

> While one thread holds the GIL, any other thread can call PyObject_Free
> with a pointer that was returned by the system malloc.

What _was_ supported was more generally that

    any number of threads could call PyObject_Free with pointers that were
    returned by the system malloc/realloc

at the same time as

    a single thread, holding the GIL, was doing anything whatsoever (including
    executing any code inside obmalloc.c)

Although that's a misleading way of expressing the actual intent; more
on that below.

> The following situation is *not* supported:
>
> While one thread holds the GIL, another thread calls PyObject_Free with
> a pointer that was returned by PyObject_Malloc.

Right, that was never supported (and I doubt it could be without
introducing a new mutex in obmalloc.c).

> I'm hoping that I got things a little better this time around. I've
> submitted my updated patch to the patch tracker. For reference, I've
> included links to SourceForge and the previous thread.
> 
> Thank you,

Thank you!  I probably can't make time to review anything before this
weekend.  I will try to then.  I expect it would be easier if you
ripped out the horrid support for PyObject_Free abuse; in a sane
world, the release-build PyMem_FREE, PyMem_Del, and PyMem_DEL would
expand to "free" instead of to "PyObject_FREE" (via changes to
pymem.h).

IOW, it was never the _intent_ that people be able to call
PyObject_Free without holding the GIL.  The need for that came from a
different problem, that old code sometimes mixed calls to PyObject_New
with calls to PyMem_DEL (or PyMem_FREE or PyMem_Del).  It's for that
latter reason that PyMem_DEL (and its synonyms) were changed to expand
to PyObject_Free.  This shouldn't be supported anymore.

Because it _was_ supported, there was no way to tell whether
PyObject_Free was being called because (a) we were catering to
long-obsolete but once-loved code that called PyMem_DEL while holding
the GIL and with a pointer obtained by PyObject_New; or, (b) somebody
was calling PyMem_Del (etc) with a non-object pointer they had
obtained from PyMem_New, or from the system malloc directly.

It was never legit to do #a without holding the GIL.  It was clear as
mud whether it was legit to do #b without holding the GIL.  If
PyMem_Del (etc) change to expand to "free" in a release build, then #b
can remain clear as mud without harming anyone.  Nobody should be
doing #a anymore.  If someone still is, "tough luck -- fix it, you've
had years of warning" is easy for me to live with at this stage.

I suppose the other consideration is that already-compiled extension
modules on non-Windows(*) systems will, if they're not recompiled,
continue to call PyObject_Free everywhere they had a
PyMem_Del/DEL/FREE call.  If such code is calling it without holding
the GIL, and obmalloc.c stops trying to support this insanity, then
they're going to grow some thread races they woudn't have if they did
recompile (to get such call sites remapped to the system free).  I
don't really care about that either:  it's a general rule that
virtually all Python API functions must be called with the GIL held,
and there was never an exception in the docs for the PyMem_ family.

(*) Windows is immune simply because the Windows Python is set up in such
    a way that you always have to recompile extension modules when Python's
    minor version number (the j in i.j.k) gets bumped.