[Python-Dev] Mixing memory management APIs

[Tim Peters]

[NeilS, growing older but wiser, embraces the wisdom of giving up <wink>]

[Michael Hudson]
> And then change all the current uses of PyObject_Del to the new API?

It would mean changing all uses of all memory macros in the core to use new
macros.

> What would that buy us?

The possibility to move to Vladimir's malloc implementation without breaking
any extension modules (none:  no breakage at all).  I want the Python core
to use Vladimir's malloc/free, and until the fabled free-threading gets
implemented, to use a version that *exploits* the GIL to eliminate
malloc/free lock overhead too.  We know for a fact that some major extension
modules misused the existing memory API (via mismatching "get memory"->"free
memory" pairs), and it's so Byzantine and ill-documented that this shouldn't
be a surprise.

Beyond that, I don't believe we've ever said anything about thread safety
wrt the existing memory API, simply because we relied on the platform
malloc/free to provide thread safety even in the worst of cases.  But if the
Python core switches to a gimmick that relies on the GIL, then even
extensions that use the current API properly (wrt correct matching pairs)
may get into huge trouble if the underlying allocator stops doing its own
layer of locking.

The intent with new macros would be to spell out the rules.  Extensions that
wanted to play along could switch, while extensions that ignored the issues
would continue to work with the existing seven ways to spell "malloc" (and
seven to spell "free").

> ...
> I don't what we have is so bad; a helpful tip is that if you're using
> the _Free/_FREE/_Malloc/_REALLOC/etc interfaces, stop.  That gets rid
> of half the problem.

But only for extensions that are actively maintained by people who are keen
to dig into how they've abused the current API.  It's likely easier for the
core to move to new macros than to fully debug even one large extension
module that's been working so far by luck.

a-big-hammer-wouldn't-be-called-that-if-it-weren't-big<wink>-ly y'rs  - tim