[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit

[Tim Peters]

Tim Peters <tim at python.org> added the comment:

BTW, the phrase "missing tp_traverse" is misleading.  If an object with a NULL tp_traverse appears in a gc generation, gc will blow up the next time that generation is collected.  That's always been so - gc doesn't check whether tp_traverse is NULL, it just calls it.  It's tp_clear that it checks, because that one is optional.

I don't recall any problem we've had with extensions implementing the gc protocol incorrectly or incompletely.  It's this issue's problem:  containers not participating in gc _at all_.

If we had a traditional mark-sweep collector, that would be massively catastrophic.  Miss a pointer and you can conclude a live object is actually trash, and tear it down while it's still in use.

Our problem is the opposite:  miss a pointer and we can conclude a trash object is actually live.  At the start, the worst that _could_ do is leak memory.  It's the later introduction of time-to-die finalizers (weakref callbacks at first) that created the opportunity for segfaults:  the only 100% clearly safe way to run finalizers in cyclic trash is to force them to run _before_ anything at all is torn down by force (tp_clear).  But to run them in advance, we have to know the relevant objects _are_ trash.  Which we can't always know if containers don't always participate.

While Neil & I haven't thought of ways that can go wrong now beyond that a "surprise finalizer" may get run any number of times, that doesn't mean far worse things can't happen - just that they'll surprise us when they do :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38006>
_______________________________________