[Python-Dev] Memory Allocator Part 2: Did I get it right?

[Evan Jones]

On Feb 15, 2005, at 17:52, Tim Peters wrote:
> As I said before, I don't think we need to support this any more.
> More, I think we should not -- the support code is excruciatingly
> subtle, it wasted plenty of your time trying to keep it working, and
> if we keep it in it's going to continue to waste time over the coming
> years (for example, in the short term, it will waste my time reviewing
> it).

I do not have nearly enough experience in the Python world to evaluate 
this decision. I've only been programming in Python for about two years 
now, and as I am sure you are aware, this is my first patch that I have 
submitted to Python. I don't really know my way around the Python 
internals, beyond writing basic extensions in C. Martin's opinion is 
clearly the opposite of yours.

Basically, the debate seems to boil down to maintaining backwards 
compatibility at the cost of making the code in obmalloc.c harder to 
understand. The particular case that is being supported could 
definitely be viewed as a "bug" in the code that using obmalloc. It 
also likely is quite rare. However, until now it has been supported, so 
it is hard to judge exactly how much code would be affected. It would 
definitely be a minor barrier to moving to Python 2.5. Is there some 
sort of consensus that is possible on this issue?

>> While one thread holds the GIL, any other thread can call 
>> PyObject_Free
>> with a pointer that was returned by the system malloc.
> What _was_ supported was more generally that
>
>     any number of threads could call PyObject_Free with pointers that 
> were
>     returned by the system malloc/realloc
>
> at the same time as
>
>     a single thread, holding the GIL, was doing anything whatsoever 
> (including
>     executing any code inside obmalloc.c)

Okay, good, that is what I have assumed.

> Although that's a misleading way of expressing the actual intent; more
> on that below.

That's fine. It may be a misleading description of the intent, but it 
is an accurate description of the required behaviour. At least I hope 
it is.

>   I expect it would be easier if you
> ripped out the horrid support for PyObject_Free abuse; in a sane
> world, the release-build PyMem_FREE, PyMem_Del, and PyMem_DEL would
> expand to "free" instead of to "PyObject_FREE" (via changes to
> pymem.h).

It turns out that basically the only thing that would change would be 
removing the "volatile" specifiers from two of the global variables, 
plus it would remove about 100 lines of comments. :) The "work" was 
basically just hurting my brain trying to reason about the concurrency 
issues, not changing code.

> It was never legit to do #a without holding the GIL.  It was clear as
> mud whether it was legit to do #b without holding the GIL.  If
> PyMem_Del (etc) change to expand to "free" in a release build, then #b
> can remain clear as mud without harming anyone.  Nobody should be
> doing #a anymore.  If someone still is, "tough luck -- fix it, you've
> had years of warning" is easy for me to live with at this stage.

Hmm... The issue is that case #a may not be an easy problem to 
diagnose: Some implementations of free() will happily do nothing if 
they are passed a pointer they know nothing about. This would just 
result in a memory leak. Other implementations of free() can output a 
warning or crash in this case, which would make it trivial to locate.

> I suppose the other consideration is that already-compiled extension
> modules on non-Windows(*) systems will, if they're not recompiled,
> continue to call PyObject_Free everywhere they had a
> PyMem_Del/DEL/FREE call.

Is it guaranteed that extension modules will be binary compatible with 
future Python releases? I didn't think this was the case.

Thanks for the feedback,

Evan Jones